3D spatial mapping in the 3D coordinate system of an AR headset using 2D images

Abstract

Techniques for generating 3D spatial mapping are described. One operation can be receiving a 2D image of a patient's anatomy from a medical imaging device. Another operation can be identifying the placement of the imaging device in a 3D coordinate system of an augmented reality (AR) headset when capturing the 2D image. Graphical marks can be received from a user for points within the person's body in the 2D image. 3D coordinates of the points in the 3D coordinate system can be calculated using the position and orientation of the imaging device relative to the AR headset and the depth of the points to generate the 3D spatial mapping.

Description

[Technical Field]

[0001] Mixed reality or augmented reality is an area of ​​computing technology in which images from the physical world and virtual computing worlds can be combined into a mixed reality world or view. In mixed reality, people, places, and objects from the physical world and virtual constructs become an integrated visual environment for a user of a mixed reality device. Mixed reality experiences can be delivered through existing commercial or custom software, along with the use of VR (Virtual Reality) or AR (Augmented Reality) headsets or devices.

[0002] Augmented reality (AR) is an example of mixed reality in which a live, direct (i.e., real) or indirect view of a physical, real-world environment is augmented or supplemented with computer-generated sensory input, such as images, audio, video, graphics, or other data. Such augmentation can be implemented so that real-world locations are viewed in relation to environmental elements. With the aid of augmented AR techniques (e.g., adding computer vision and object recognition), information about the real world around a user can become interactive and digitally modified (e.g., via computer graphic overlays).

[0003] A problem faced by AR systems or AR headsets is identifying the position and orientation of objects with high accuracy. Similarly, aligning the position of virtual elements with a live view of the real-world environment can be a challenge. For example, aligning a virtual object with a physical object when viewed through an AR headset can be difficult because the luminescence or light of the virtual object tends to obscure the underlying physical object with which the virtual object is to be aligned. While aligning a virtual object with a physical object to within approximately a few centimeters can be useful for entertainment and less demanding applications, greater resolution in the positioning and alignment of AR systems may be desirable in scientific, engineering, and medical fields. As a result, the positioning and alignment process for AR used in scientific, engineering, and medical fields can be performed manually, which can be time-consuming, cumbersome, and inaccurate. [Brief explanation of the drawings]

[0004] [Figure 1A] FIG. 1 illustrates an example of mapping or registering 2D images (e.g., ultrasound images) to a human body to create a 3D spatial mapping. [Figure 1B] 1B shows an example of a 2D ultrasound image that may be captured for FIG. 1A. [Figure 1C] 1 shows an example of a 2D ultrasound image superimposed on a person's body using an AR headset at a location defined by the location and orientation of the ultrasound transducer. [Figure 2A] 1 illustrates an example of points of interest identified using at least two 2D ultrasound images to form a trajectory or guidance path. [Figure 2B] An example is shown in which a user may sweep over an anatomical structure desired to be identified using a combination of ultrasound images. [Figure 2C] 1 shows an example of two points that can be marked on two views of a common anatomical structure or location. [Figure 3A] 1 illustrates an exemplary arrangement of 2D images (e.g., X-ray generated images) that may be registered with a human body. [Figure 3B] 3B shows an example of 2D images that can be further registered by using points in the two different images of FIG. 3A. [Figure 4A] 1 shows an example of a first 2D image (eg, an X-ray generated image) of a person's body that may be captured using a C-arm or medical imaging device. [Figure 4B] 4B shows an example of a second 2D image captured by a medical imaging device from a second position compared to FIG. 4A. [Figure 4C] 1 illustrates an example of a first 2D image and a second 2D image displayed together using an AR headset. [Figure 5A] An example method is presented for displaying X-ray generated images while viewing a patient's anatomy using an augmented reality (AR) headset. [Figure 5B] 1 shows orthogonal side views of a first 2D image and a second 2D image when intersected by a line defined by a user or medical professional. [Figure 5C] 10 illustrates exemplary lines that may be marked in an X-ray generated image having a non-parallel view. [Figure 5D] 1 shows an exemplary perspective view through an AR headset of two X-ray generated images intersecting at a point or line in each image. [Figure 5E] FIG. 10 shows another example perspective view of two X-ray generated images intersecting at a line in each image as viewed through an AR headset. [Figure 6] 1 shows an example of a 2D image or X-ray generated image that can be overlaid onto an anatomical structure. [Figure 7] 1 shows an example of one 2D image or X-ray generated image displayed as an overlay on an anatomical structure. [Figure 8] An example is shown in which a medical professional guides a needle into a simulated skull. [Figure 9] 10 shows an example of a second perspective view of a user using an AR headset to guide a needle into a simulated patient's skull. [Figure 10] An example of a system for aligning X-ray generated images with a patient's anatomy using an augmented reality (AR) headset is shown. [Figure 11] 1 shows an example of an x-ray generated image that has been shifted and scaled to match the anatomical scale of a patient's anatomy. [Figure 12A] It is shown that the first 2D x-ray generated image can be a projection through the anatomy. [Figure 12B] 10 shows an example of a second 2D image that may be captured of an anatomical structure. [Figure 12C] 1 shows an example of a first x-ray generated image and a second x-ray generated image overlaid on an anatomical structure. [Figure 13A] An example of how a first object can appear small in a 2D X-ray image when the image is reduced to match the size of a second object is shown. [Figure 13B] It shows that a corner can be moved down within the X-ray projection field, allowing a portion of the image captured of the first object to be enlarged. [Figure 13C] Illustrates the use of a point in the first x-ray generated image and a line in the second x-ray generated image. [Figure 14] 1 is a flowchart illustrating an exemplary method for creating a 3D spatial mapping or 3D spatial guide. [Figure 15] 1 is a flowchart illustrating an example of a method for 3D spatial mapping. [Figure 16] An exemplary method for aligning X-ray generated images with human anatomy using an augmented reality (AR) headset is presented. DETAILED DESCRIPTION OF THE INVENTION

[0005] Reference will now be made to the exemplary embodiments illustrated in the drawings, and specific language will be used herein to describe the same. It will nevertheless be understood that no limitation of the scope of the technology is thereby intended. Variations and further modifications of the features shown herein, and additional uses of the embodiments shown herein that will occur to those skilled in the art in possession of this disclosure, should be considered within the scope of this specification.

[0006] The present technology may be used to acquire one or more 2D images from a medical imaging device (e.g., an X-ray device, an ultrasound device, etc.), the position and orientation of the image may be acquired, and one or more points, lines, or 3D regions may be identified on the image. These identified points may be points, lines, 2D regions, or 3D regions identified and marked by a user, or the identified points may be points, lines, 2D regions, or 3D regions identified by a machine. The points, lines, or 3D regions may be referenced and added to the 3D coordinate space of an augmented reality (AR) headset. Identifying where a reference point, line, 2D region, or 3D region is located within the 3D coordinate space of the AR headset may be calculated by determining the position and orientation of the medical imaging device that captured the 2D medical image. Additionally, the depth of the reference point, line, or region relative to the person's body or the image source may also be determined. A user of the AR headset can then view the marked point, line, or 3D region within the 3D coordinate system of the AR headset and use the point, line, or region for navigation (e.g., navigation of a medical instrument) during a medical procedure.

[0007] In one configuration of the present technology, multiple 2D images can be acquired and the intersections of the 2D images can be aligned with each other. More specifically, the positions and orientations (e.g., placement) of the image views can be recorded. Points or lines in the two views can then be aligned with each other, providing the intersection of the two 2D images at a point, line, 2D region, or 3D region. If the intersection of the two images is a line, the line can provide a trajectory for a medical professional to navigate along or around during a medical procedure. Similarly, a medical professional may be able to navigate a medical instrument to a point or line representing a common anatomical structure in both 2D images. Because the AR headset records the points, lines, or 3D regions in the 3D coordinate system of the AR headset, these reference points can be visible in the 3D space of the AR headset. Thus, multiple images that intersect with each other by points, lines, or regions can be used to navigate in augmented reality.

[0008] Previous navigation using augmented reality (AR) has used 3D datasets such as MRI (Magnetic Resonance Imaging) or CT (Computed Tomography) scans. In contrast, this technology uses one or more 2D images to enable more precise navigation to anatomical structures within a patient's body.

[0009] 1A illustrates a system and / or method for using techniques to create 3D spatial mapping. In one operation, a 2D image 110 of a patient's anatomy 120 may be received, which may represent a projection from a medical imaging device 112. The medical imaging device 112 may be an ultrasound device or another medical imaging device that provides a 2D image and includes or can determine a known depth for the 2D image.

[0010] The placement (i.e., position and orientation) of the imaging device 112 in the 3D coordinate system of the augmented reality (AR) headset may be identified by the AR headset when the 2D image is captured. The placement of the imaging device 112 may define a position and angular orientation or other spatial information for the imaging device 112 (e.g., in the 3D coordinate space or Cartesian space of the AR headset). The placement of the imaging device 112 may be determined by the AR headset, which may scan an optical code 114 or other marker on the imaging device 112, an ultrasound transducer, or an ultrasound image capture device.

[0011] Using one or more markers or optical codes (e.g., a 2D barcode, a linear barcode, or an April code), the AR headset can determine the position and / or orientation of the imaging device 112. Once the optical code 112 is captured, certain aspects about the ultrasound image are known, including the position and orientation of the ultrasound image relative to the imaging device. This allows the ultrasound image received from the imaging device to be passed to the AR headset along with this additional image information. Because the location and position of the imaging device 112 is known in the 3D coordinate system of the AR headset, the AR headset can receive the 2D image and orient it in space at the correct position and orientation for the edge of the imaging device 112 that is in contact with the person's skin.

[0012] In ultrasound imaging, the depth of an anatomical structure is available from the ultrasound imaging device, and the orientation and position of the image can be obtained using markers (e.g., optical codes) on the ultrasound device. Using ultrasound imaging, a point in the 3D coordinate system of the AR headset can be identified with a single ultrasound scan. Ultrasound imaging can also be calibrated from the contact of the ultrasound transducer with the skin over the anatomical region. For example, a physician can see a vein in an ultrasound image, and the vein is measured to be 10 cm deep from the transducer point on the skin, as reported by the ultrasound device.

[0013] FIG. 1B shows a 2D ultrasound image 110 that may be captured in FIG. 1A. This image is of the liver using a curvilinear scanning device. The image may be displayed to a user or medical professional in an AR headset, on a computer workstation, on a desktop computer, on a mobile device, or using another display method. When input by a user, graphical marks 122 may be received for points representing anatomical structures in the 2D image 110 from the imaging device 112. For example, a physician may place a mark or point in the image, and the ultrasound device may know or measure the depth of the point where the graphical mark 122 was placed.

[0014] The 3D coordinates of the point can be calculated for the 3D coordinate system of the AR headset using the position of the ultrasound imager 112 relative to the AR headset and the depth of the point provided by the ultrasound imager. The ability to define the 3D coordinates of the graphical marks can be used to generate 3D spatial mapping. 3D spatial mapping using the graphical marks can be used to guide a medical professional in accessing the point of interest during a medical procedure or to guide portions of a medical procedure. For example, a physician may want to place the tip of a needle or trocar at the point marked by the physician. The physician can begin inserting the needle or trocar into the patient's body, and then, based on whether the acquired 2D images and reference points indicate that the medical tool is on the correct trajectory and will eventually reach the marked point, the physician can change the position or orientation of the medical tool as the physician is inserting the needle or trocar into the patient's body. Because the point of interest is known in the 3D coordinate system of the AR headset, the physician can view the point of interest from any angle using the AR headset. Additionally, instrument tracking allows the needle or instrument to be precisely navigated to the marked 3D coordinates.

[0015] The surface of the patient's skin may be located at the upper radius of curvature of the ultrasound image in FIG. 1B. A line representing depth is then drawn to the potential target or point 122. The ultrasound scanning device head may have any shape, such as curved, flat, or triangular. Objects on the ultrasound image have depth because the transducer is in contact with the patient and the time it takes for ultrasound waves to bounce off the anatomical object is known. When an ultrasound beam is used to capture the image, the medical professional may mark the point, and the location of the ultrasound image is captured using an optical code, as described above. This may result in a 3D reference point in the space defined by the AR headset. Once the ultrasound device is removed, the medical professional may target the point with a medical instrument or other medical procedure.

[0016] The ultrasound image can be linked to an optical code located on the person's body. The 2D image can then be moved to maintain a reference to the person originally captured, even if the patient moves. For example, if a doctor needs to insert a needle into a person's neck in the ICU, the doctor can use ultrasound to identify a point or 3D structure on the neck. The point can be displayed in the AR headset so the doctor can confirm they are reaching the correct point.

[0017] The ultrasound transducer 112 (FIG. 1A) may use an electronically steered fan beam or linear beam. If the ultrasound beam is moved or steered mechanically or electronically, this steering may be taken into account when determining the location of the ultrasound image.

[0018] FIG. 1C shows an example of an ultrasound image 110 superimposed by an AR headset at a location defined by the position and orientation of the ultrasound transducer. The location and orientation of an ultrasound marker can be detected by an optical tag 114 (or marker) on the ultrasound transducer. For example, a medical professional can identify an anatomical structure (e.g., the center of the aorta) and mark a location 122. The optical tag 114 on the ultrasound transducer can be used as a reference anchor for the ultrasound image. The point marked by the medical professional can then be defined within the 3D coordinate space of the AR headset. This point in space can remain defined for the medical professional, but the medical professional can use the point to insert a needle into the patient, insert the needle into the aorta, perform a biopsy, etc. This guidance can be performed using a point on the ultrasound image, data collected from the optical tag on the ultrasound device, and the depth of the point reported by the ultrasound device. The AR headset can detect optical tags on the ultrasound transducer and determine the position and orientation (e.g., location using Cartesian coordinates and angular orientation) of the ultrasound image in the 3D coordinate space of the AR headset. Because the AR headset can obtain the angle, position, and depth of a point in the ultrasound image, these operations can be performed using a single ultrasound image. A medical professional can navigate to the point using a medical instrument (e.g., a needle or trocar), and the AR headset can also generate a virtual portion of the medical instrument (e.g., a virtual needle) by tracking the instrument as it is inserted into the patient.

[0019] 1A and 1B, an ultrasound image 110 may be acquired from a portion of a patient's body while a medical procedure is being performed by a medical professional using an ultrasound probe or ultrasound transducer 112. The ultrasound transducer 112 may be mobile or movable relative to the person's body. The ultrasound image or sonogram may be a 2D, 3D, or 4D (e.g., including a time series) ultrasound image produced by sound waves bouncing or echoing off body tissue within the patient's body. The echoes may be processed by a computer to create the ultrasound image or sonogram.

[0020] These ultrasound images 212 can be relatively fast and inexpensive to acquire during a medical procedure, but the resolution, accuracy, and localization of ultrasound images may not be as advanced as other types of imaging, such as CT scans, MRIs, and other imaging modalities. The present technology provides the ability to localize or reference the ultrasound image 110 in the 3D coordinate system of the AR headset to assist medical professionals during a medical procedure. The ultrasound image 110 projected onto the patient in the AR headset may be partially transparent (e.g., using a transparency value set by the medical professional), or the ultrasound image in the AR headset may be opaque.

[0021] The ultrasound image 110 or sonogram may also be combined with an image dataset (e.g., an MRI (magnetic resonance imaging), a CT scan, etc.), which may be more accurate, clearer, have higher resolution, larger, and have better tissue contrast information compared to the ultrasound image 110. An optical code on a person's body may be attached to an image visible marker. This image visible marker may enable alignment of an image dataset previously acquired from the patient with the patient's body. Thus, a medical professional can view the ultrasound image 114 combined with the high-resolution image dataset through an AR headset, aligned and projected to the correct position on the patient's body. For example, when a medical professional is performing a medical procedure on a patient's liver using ultrasound equipment, a limited portion of the patient's liver can be viewed at any time using the ultrasound image, but the entire liver can be viewed using an image dataset acquired together with the ultrasound image, such as a CT scan, an MRI scan, or a PET scan. The ultrasound image and the image dataset can be co-localized in the 3D space viewed by the AR headset. Details regarding the use of markers, optical codes, and image-visible markers to align image data sets and images with a person's body are further described in U.S. Patient No. 11,287,874 to Gibby, entitled "USING OPTICAL CODES WITH AUGMENTED REALITY DISPLAYS," U.S. Patient No. 10,825,563 to Gibby, entitled "ALIGNING IMAGE DATA OF A PATIENT WITH ACTUAL VIEWS OF THE PATIENT USING AN OPTICAL CODE AFFIXED TO THE PATIENT," and U.S. Patient No. 10,010,379 to Gibby, entitled "AUGMENTED REALITY VIEWING AND TAGGING FOR MEDICAL PROCEDURES," each of which is incorporated herein by reference in its entirety.

[0022] In the past, the ultrasound image 110 was not fixed to a reference point relative to the patient marked on the 2D image. Using the optical code as described above provides a reference point in the 3D coordinate space viewed by the AR headset. Furthermore, the markers (i.e., the optical code) may be used to align the ultrasound image 110 with one or more other image datasets.

[0023] When a medical professional attempts to perform a medical procedure such as a breast biopsy, the medical professional may use ultrasound equipment, but it may be difficult to see the actual lesion in the ultrasound image. However, the lesion may be visible in a CT, MRI, PET, nuclear, or other image dataset that is displayed in combination with or co-localized with the ultrasound image. Thus, ultrasound images may be used to provide images captured during the procedure (e.g., in real time), while previously captured detailed anatomical structures may be simultaneously referenced using an image dataset with higher spatial or contrast resolution.

[0024] The ultrasound transducer 114 may pass through the surface of the body or may be inserted into an orifice in the body. This may allow fusion or composite views of the ultrasound image and the image dataset to provide many different composite views of the patient's body. The registration of the real-time image with the patient's body and the image dataset may be applied to any type of real-time (e.g., video) medical imaging where the position and orientation of the real-time image may be obtained from an imaging device transducer, emitter, or detector. Additional examples of such real-time imaging that may be substituted for ultrasound images are CT or fluoroscopic images.

[0025] The AR headset has semi-transparent lenses so that the medical professional performing the procedure can view 2D images from a medical imaging device while viewing the patient's anatomy through the augmented reality (AR) headset.

[0026] A procedure guidance indicator, such as a graphical line, can be generated by the AR headset and used to guide a medical tool identified by the AR headset to the 3D coordinates of a point in the 2D image dataset. The procedure guidance indicator can be another graphical user interface element, such as a graphical arrow, a path indicator, an animation, or other guidance graphic.

[0027] FIG. 2A shows an example of a first 2D image 202 that can be used together with a second 2D image 204 from an ultrasound imaging device. Points of interest in each 2D ultrasound image can create a guided path 224 or pathway that can be followed by a medical instrument controlled by a medical professional. This guided path 224 can provide a trajectory in the 3D coordinate system of the AR headset for the physician to follow. A first graphical mark 220 can also be placed on the first 2D image 202 to mark a first point on the path, and a second graphical mark 222 can be placed on the second 2D image 204. Displaying two ultrasound images and their respective graphical marks can help create a 3D spatial mapping (e.g., a guided path) or a 3D spatial reference guide for a medical professional while performing a procedure. Additionally, a portion of an anatomical structure or a cross-section of an anatomical structure can be imaged (e.g., a vein wall, a duct, a blood vessel, a bone, etc.) and then marked and used as a guide point for the guided path.

[0028] A navigation trajectory can be created by acquiring ultrasound images from two different positions (non-parallel images) or two different probes. Points can be marked on each of two or more ultrasound images, and the ultrasound instrument itself has depth and directionality. The ultrasound beam can be calibrated so that the ultrasound device can provide the depth of a location within the ultrasound image. Furthermore, markers or optical codes 230 on the ultrasound device or the shape of the ultrasound transducer can be used to determine the position and / or orientation of the ultrasound transducer. Thus, each marked point or location is known to the AR device within a 3D coordinate system. The two points can create a trajectory that can be navigated by a physician using a medical instrument. Figure 2A shows an example of creating a portal vein (PV) to hepatic vein (RTHV) shunt, such as in a DIPS procedure (direct intrahepatic portacaval shunt).

[0029] 2B shows that a medical professional, physician, or user may use ultrasound images to sweep over an anatomical structure desired to be identified. For example, ultrasound images may be used to find blood vessels where dark flow voids are visible, or a physician may color the blood vessels, thereby encoding the velocity of arterial or venous blood vessels.

[0030] A portion of the anatomical structure can be identified in each ultrasound image as the ultrasound image is captured. When a medical professional sweeps the ultrasound transducer 112 along the anatomical structure, as indicated by arrow 260, the anatomical structure (e.g., a blood vessel) will have multiple ultrasound images 250, 252, 254, 256 captured along the sweep. Although four images are shown, tens, hundreds, or even thousands of images may be captured in a single sweep. Thus, a cross-section of the tubular structure (e.g., a blood vessel) can be identified in each of the images.

[0031] The anatomical structures identified in each image can then be converted into virtual structures representing blood vessels. In other words, a virtual model can be created for the anatomical structure of interest, whether it be a blood vessel, kidney, liver, or the like. This virtual modeling can result in a structure in 3D coordinate space for AR headset guidance. Furthermore, the medical professional can sweep a portion of the anatomical structure to create an outline of the structure and volume of the anatomical structure. In one configuration, the medical professional can mark points on the blood vessels, which can be incorporated into building the virtual model.

[0032] In one example, the virtual model can be automatically generated using automated methods. More specific automated methods for detecting anatomical structures can include thresholding, signal intensity detection, edge detection, or artificial intelligence to detect anatomical structures, and the system can be trained to identify specific types of anatomical structures (e.g., blood vessels, liver, kidneys, etc.).

[0033] Sweeping an ultrasound transducer over an anatomical structure may enable a user or physician to navigate to the structure during a medical procedure after imaging. The physician may sweep through the anatomical structure and capture many ultrasound images. The physician may then have AI detect the anatomical structure and build a virtual model of the anatomical structure. The system recognizes the depth of the anatomical structure due to the ultrasound-detected depth, and the generated virtual anatomical structure may be used by the physician to navigate relative to the anatomical structure during the procedure using an AR headset. For example, if a physician is attempting to insert a needle into an artery in the groin, the physician may sweep an ultrasound sensor or detector along the artery. This may generate a 3D target whose shape and depth the physician can see through the AR headset.

[0034] Alternatively, the physician may use markings on the ultrasound image to assist the computer AI in identifying the anatomical structures.

[0035] It is useful to be able to identify points, lines, 2D regions, or 3D regions as targets within a patient. For example, lines can be used as navigation paths, or 3D anatomical structures can be used to navigate to or around structures or lesions to avoid damage. When x-ray generated images are used, multiple images can be used to find image intersections. This method involves identifying where the imaging source is, or the source of the x-ray generated images, and the location of this x-ray source provides the beam trajectory. Identification of the source location and orientation can also be identified from the device itself (e.g., in DICOM information). Alternatively, location and orientation information can be obtained from an optical code on the C-arm or gantry of the x-ray device. Furthermore, the location and orientation of the x-ray generated images can be obtained using markers, tags, or optical codes with image-visible markers in the x-ray generated images, and the location and orientation of the x-ray generated images can be extrapolated back to the source and determined using triangulation.

[0036] Once the location of the image source has been identified, common anatomical structures or landmarks can then be identified in the patient. This provides reference points, lines, or regions for registering the x-ray-generated images using the patient. The fiducial markings can be obtained from markings manually made by a medical professional, or the fiducial markings can be detected by a device using machine learning and machine vision. For example, the device can identify the pedicles of a vertebra. The markings can be lines (e.g., instrument trajectories) so that the lines follow the shape of the bone, or the markings can be 3D structures.

[0037] These markings may be on two images from two non-parallel projections. The intersection of these markings may be calculated between the two X-ray generated images, which provides one or more locations in the 3D coordinate space of the AR headset. Identifying the location of the X-ray source and identifying an integrated point, line, or 3D structure on a common anatomical structure in each image allows for the calculation of the 3D coordinates of the target point, line, or marked area. The 3D coordinates allow for targeting of the anatomical structure using a 3D guidance system in the AR headset.

[0038] The system can calculate where the X-ray source is, or the origin of the 2D image, using an optical code on the imaging device. In another configuration, the imaging source may have a coordinate reference system that uses sensors to report the location in space of the imaging device (e.g., the device uses a potentiometer to communicate the source of the beam). Alternatively, an image visible code may be identified on the patient, and the system can then back-project to the source from the image visible code found in the 2D image. This technology can identify the patient's anatomical structure in 3D coordinate space when viewed through an AR headset. The target can be a point, line, or path, a 2D area, or a 3D structure.

[0039] FIG. 2C shows an example of two points that may be marked on two views of a common anatomical structure or location (in this case, the location of the medial branch). A first point 570 (e.g., represented by a circle in this illustration) is marked on a first view of the patient's spine. This point may represent an anatomical structure of interest to a medical professional. Additionally, a second point 570 may be marked on a second view of the spine. The first point and the second point may mark a common anatomical structure. As a result, the first point and the second point may be aligned with a navigation point in the 3D coordinate system of the AR headset (as described elsewhere herein).

[0040] 3A shows an arrangement of 2D images 310, 312 that may be aligned with a person's body. Multiple 2D images 310, 312 of the body or anatomy of a person or patient may be received or identified. The images of the patient's anatomy may represent non-parallel projections from an imaging device having an X-ray emitter 316 and an X-ray detector 314. The 2D images are projections of the person's body, and the depth of the anatomy relative to the X-ray detector 314 may vary depending on the person's placement, so the 2D images do not have depth associated with them. In one example, the 2D images may be X-ray, ultrasound, or other projected medical images.

[0041] The placement of the imaging device can be identified in the 3D coordinate system of the augmented reality (AR) headset when capturing multiple 2D images. For example, the position and orientation of the imaging device's X-ray detector 314 or X-ray emitter 316 can be detected using pattern recognition to identify the shape of the imaging device or using at least one optical code 320 on the imaging device. The device's position also defines the location of the X-ray emitter 316 and the path of the X-rays 350 from the X-ray emitter.

[0042] At least two graphical marks 322, 324 may be received from a user for a common anatomical structure in the multiple 2D medical images. The graphical marks may be received using a graphical interface of an AR headset. Alternatively, the graphical marks may be placed on the images using a workstation, tablet, mobile device, or other separate computing device. The graphical marks may be a point, a line, a navigation path, a 2D area, a 3D area target, or another graphical mark located on the common anatomical structure on each 2D medical image. Portions of the graphical marks may be aligned, but the graphical marks need not be identical. The first graphical mark 322 may be located on the first 2D medical image 310, and the second graphical mark 324 may be located on the second 2D medical image 324. These graphical marks may be located on the common anatomical structure in each 2D image. For example, the graphical mark may be a dot located on the iliac crest, a portion of a bone, an anatomical landmark, an air collection, a foreign body, or another identifiable anatomical structure visible in each 2D image.

[0043] The 3D coordinates within the AR headset representing the common anatomical structure can be determined within the 3D coordinate system (or space) of the AR headset. The 3D coordinates of the common anatomical structure of two or more graphical marks in the 2D image can be calculated by extrapolating a line from the projection source (e.g., the X-ray emitter 316 treated as a point source) to at least two graphical marks in the non-parallel projection to form an intersection of at least two lines to define the 3D coordinate. The mathematical point where the two lines intersect is the depth or coordinate in 3D space where the common anatomical structure is located. This allows the patient's anatomy to be used to align the two non-parallel 2D images with each other. The 2D image can then be overlaid at its location on the anatomy representing the depth of the image as viewed by the AR headset.

[0044] 3D spatial mapping can further be generated by creating this overlay by displaying two 2D images provided by a medical imaging device while viewing the patient's anatomy using an augmented reality (AR) headset. This means that a medical professional using the overlay or precisely aligned 2D images can have a spatial reference in more than one axis for performing a medical procedure on a patient using a medical instrument. For example, 3D spatial mapping or reference images can be provided for a medical procedure in at least two axes (e.g., or as many axes as the 2D images were captured).

[0045] Additionally, a navigation path, which is a graphical reference, can be provided from the identified medical tool by the AR headset to the 3D coordinates of common anatomical structures. The navigation path can be a line, curve, or other path that can be generated using the AR headset and superimposed on the patient's body.

[0046] An optical code having image visible markers 460 in or on the marks may be used to identify the position and orientation of a person's body and may be recorded in an X-ray image as image visible markers 358. As described above, image visible markers 460 may be used to more accurately align an X-ray image with the patient's body, or image visible markers may be used to align images from other modalities (e.g., CT, MRI, etc.) with the body when viewed through an AR headset.

[0047] FIG. 3B further illustrates that 2D images can be registered using points in two different images of FIG. 3A. For example, a medical professional can view an X-ray and pinpoint the same anatomical location or corresponding point of interest on two X-ray-generated images. Both images can be in different planes. For example, the images can be perpendicular to each other or at another angle. Thus, the anatomical structure is calculated to be at a specific point in 3D space relative to the AR headset. The medical professional can guide a needle to that point using only the two 2D X-ray-generated images. Each X-ray-generated image can be within its own virtual pyramid representing the projection of the X-ray-generated image. Furthermore, a virtual line 350 can be placed on the common anatomical structure in each of the 2D images. The 2D images can be set to intersect on that or another user-defined line at the same location in each 2D image. As seen in FIG. 3B, the 2D images can intersect at an angle, and this difference in angle between the 2D images can result in a 3D spatial mapping that can be viewed by the medical professional. For example, a physician may use a needle in a medical procedure and the physician can see as the needle moves up or down toward the correct point or line in a plane. Two points of interest may create a guided path or pathway that the medical instrument can follow. This line may provide a 3D trajectory for the physician to follow. The 2D image may be an X-ray generated image, an ultrasound image, a CT generated image, or an MRI generated image.

[0048] This aspect of the technology uses two planar images to find a 3D target in space for an AR headset. The patient's own anatomy can be used by a medical professional or machine methods to mutually align the images. Mutual alignment can be performed by obtaining spatial information from the image source so that the images can be aligned to known spatial information about the patient in the 3D coordinate system of the AR headset. The use of two or more image views for X-ray-generated images (or images without measured depth) provides 3D navigation.

[0049] The images may be x-rays, ultrasound images, fluoroscopic images, or other planar medical images. On the patient images, common anatomical structures may be identified in two images that are non-parallel projections. The patient anatomical structures may be identified using points, lines, 2D regions, or 3D regions.

[0050] Images can be referenced for their spatial location using different methods. One method can use image visible codes on a person's skin, which provide some of the spatial information. Information about how two images relate to each other can then be provided by the user or by image detection. If the images are at 90 degrees or 30 degrees to each other, a change in the angle between the images will also change where the intersection point between the images is. Thus, orientation angle data can be obtained from image visible tags on the patient, which can be used to calculate the angle at which the image was captured by the medical imaging device.

[0051] Alternatively, the optical tag may be on an X-ray or fluoroscopic device, and the optical tag may provide a reference for the position and orientation of the X-ray imager relative to the patient. Alternatively, the C-arm of a fluoroscope (e.g., for angiographic imaging) may have an angle provided from an internal sensor, which may then be coded into DICOM data representing the projection angle of the captured image, and the device may transfer this position and / or orientation information to the AR headset over a local network or communicate it to the AR headset by other means. This information may indicate how multiple images relate to each other.

[0052] 4A shows a first 2D image 402 (e.g., an X-ray generated image) of a body of a person 406 that may be captured using a C-arm 414. The C-arm may have an X-ray emitter 412 and an X-ray detector 416. The person's body 406 may also have an optical code 408, 430 with image-visible markers, and the 2D image may be linked to the optical code; when the person's body moves with the optical code, the optical code and image-visible code may be used as a reference for moving the 2D image within the display of the AR headset. A medical instrument 410 may be used during a medical procedure and may be included in the 2D image 402.

[0053] 4B shows a 2D image 420 captured by a medical imaging device from a second position. For example, the medical imaging device may be a C-arm 414 capable of capturing X-ray generated images. The medical imaging device may have an X-ray emitter 412 and an X-ray detector 416 set at a different position and / or orientation to capture additional 2D images or additional 2D X-ray projections compared to the first orientation of the C-arm (FIG. 4A) used to capture the first 2D image of the body of the person 406.

[0054] FIG. 4C shows an example of a first 2D image 402 and a second 2D image 430 displayed together using an AR headset. The 2D images are displayed at the same position and orientation relative to the patient's body as the 2D images were captured. The first 2D image 402, in this example, may be at a 90-degree angle from the second 2D image 420. However, any angle may exist between the first 2D image 302 and the second 2D image 420, as defined when the 2D images are captured separately. As previously mentioned, 3D spatial mapping or reference images can serve as a guide for 3D coordinate space in at least two axes during a medical procedure. A medical professional can use the 3D spatial mapping to guide a medical instrument 410 and view the 3D spatial mapping, which can help guide the medical professional as to where the instrument is relative to the tissues of the person's or patient's body.

[0055] To reiterate, some of the aforementioned configurations may use at least two images, which may be X-ray generated images, fluoroscopic images, ultrasound images, angiographic images, real-time X-ray video, etc. Both the X-ray machine and the ultrasound transducer may have one or more optical tags on the device that give their relative position and orientation in space when the 2D images are captured. However, when two non-parallel X-ray generated images are used, no code is required on the X-ray machine or ultrasound transducer, as a code on the patient allows the X-ray beams to be triangulated and determined. Points, lines, two lines, 2D regions, or 3D regions on each 3D image may then be used to identify common anatomical structures. The intersection of the two images provides a 3D coordinate structure within the AR headset that the medical professional can navigate or navigate around.

[0056] A medical professional can acquire two X-ray-generated images of a patient, which can reference an image-visible code on the patient. This reference can indicate where the 2D image is located relative to the patient. If only a single point is marked on each X-ray-generated image, the medical professional can navigate to that point in space. However, a perfect intersection of the images may not be obtainable, but a single common anatomical location can be accurately viewed through the AR headset. This method can provide a single-point target in the 3D coordinate space of the AR headset even if the images are not aligned. On the other hand, if the medical professional wants to navigate along a trajectory, a line can be used in both 2D images.

[0057] 5A illustrates an example of a method for displaying X-ray generated images 502, 504 while viewing a patient's anatomy using an augmented reality (AR) headset. Multiple X-ray generated images of the patient's anatomy may be received from an X-ray machine. Each of the X-ray generated images may represent a non-parallel positioning of the patient's anatomy relative to the X-ray machine, or a non-parallel positioning of the X-ray machine relative to the patient's anatomy. Each non-parallel positioning may define a non-parallel position and orientation of the patient's anatomy relative to the X-ray machine, or vice versa.

[0058] A first line markup 510 may be received from a user for the first X-ray generated image 502, and a second line markup 512 may be received using at least one point marking in the second X-ray generated image 504. The markups may represent common anatomical structures between the first X-ray generated image 502 and the second X-ray generated image 504.

[0059] In one configuration of this technology, a line can be drawn on a first image, and a second line can be created using a point marked on a second X-ray image. This point on the second X-ray image can form the second line by using the perspective of the AR headset. This second line, formed by implication, can then be aligned with the first line to provide an intersection between the two 2D images.

[0060] The first line 510 and the second line 512 can be aligned to generate a 3D spatial mapping using the first X-ray generated image 502 and the second X-ray generated image 504 viewable using the AR headset. The dotted line 520 indicates a line along which the first line 510 and the second line 512 can be aligned relative to a common anatomical structure. In other words, the line can be drawn in the same location on each image using the common anatomical structure as a reference. The line, present in both images, can be used by a medical professional, such as a physician performing a procedure, to guide the physician to where to place a medical tool. For example, the medical professional can follow the line as a guide path to perform the procedure (e.g., insert a needle). The medical professional knows that when the line is followed, the medical procedure is likely proceeding correctly. The medical tool can be a needle, trocar, endoscopic tool, or other medical tool.

[0061] Thus, a medical professional may have a 3D spatial reference for using a medical tool or for identifying where an incision will be made using at least two axes. This 3D reference is provided while capturing only a limited number of 2D images (e.g., 2+ images from different projection points or different viewpoints). This means that the patient does not need to have a CT scan or MRI before the medical procedure, and only capturing a limited number of X-ray images may be used. This may reduce the amount of exposure the patient receives to X-rays, magnetic fields, medical dyes, etc.

[0062] FIG. 5B shows an orthogonal edge view of the first 2D image 502 and the second 2D image 504 as they intersect at a line defined by a user or medical professional. Dot 530 represents an end view of both lines as they intersect, as seen in both 2D images along the edge to the AR headset or user. A cross-sectional outline 540 representing the likely location of the patient's bone can also be seen in FIG. 5B. This cross-section of the bone may not actually be visible to the user, but it represents how the AR headset can be used to align 2D images with actual anatomical structures. In another example, a physician could navigate the length of a bone, for example, to place an intramedullary rod.

[0063] The multiple X-ray generated images may also be aligned with the patient's anatomy viewable through the AR headset using at least one optical code and associated image visible markers depicted in the X-ray generated images. Thus, the position and orientation of the X-ray generated images may be defined in part by the position and orientation of the optical code and image visible markers viewable in the X-ray generated images by the AR headset.

[0064] The 2D image can be linked to the movement of the patient's anatomy visible in the AR headset using the optical code and associated image visible markers. As a result, the X-ray-generated image can be moved to maintain alignment of the 2D image with one or more optical codes and image visible markers on the patient's anatomy as the patient's anatomy moves away from the X-ray machine. For example, if the 2D image is linked to the optical code, then when the patient moves their arm away from the X-ray machine, the AR headset can maintain the display of the 2D image with respect to the moved arm. By using the optical code and image visible markers, the 2D image can maintain its previous orientation and position with respect to the patient's moved arm. This allows a medical professional to view the X-ray-generated image in the correct position with respect to the arm even when X-ray is not being used and X-ray-generated images or video are not being actively captured.

[0065] In one configuration, the orientation and / or position of the 2D image can be defined by the position and orientation of the imaging device capturing the 2D image. For example, the shape of the imaging device can be detected, and then the X-ray source of the image can be recognized. For example, the position of the X-ray source can be calculated using an AR headset. As a result, the way in which the 2D image should be oriented and / or positioned can also be recognized. Similarly, one or more markers can be on the imaging device, which can provide the position and orientation of the imaging device, and in turn, the position and / or orientation of the 2D image. The markers can be optical codes or other visible markers.

[0066] The position and orientation of the x-ray generated images may also be defined relative to the patient's anatomy using the geometric attributes of the x-ray projection field.

[0067] 5C shows that a line of an anatomical structure 550 may be marked on a first x-ray generated image. Additionally, a second line 552 for a common anatomical structure may be marked on a second x-ray generated image. These two lines may be aligned as discussed and illustrated in FIGS. 5A and 5B.

[0068] 5D shows a perspective view of two X-ray generated images intersecting at a point or line within each image as viewed through an AR headset. A first X-ray generated image 560 and a second X-ray generated image 562 are overlaid on a simulated patient or X-ray phantom. The first X-ray generated image 560 and the second X-ray generated image 562 may intersect at a line or point. In this example, a medical professional may use the intersection of the two images as a guidance path for a medical procedure. For example, a needle may be guided down the line where the two images intersect, or the needle may be guided to a point (or other mark) in the 3D coordinate space of the AR headset.

[0069] FIG. 5E shows a perspective view of two X-ray generated images intersecting at a line within each image as viewed through an AR headset. A first X-ray generated image 570 and a second X-ray generated image 572 may be overlaid on a simulated patient or X-ray phantom. The first X-ray generated image 570 and the second X-ray generated image may intersect at line 574. In this example, a medical professional may use the intersection of the two images as a navigation path for a medical procedure. For example, a needle 576 may be guided down line 574 at the intersection of the two images. The line 574 on each 2D image may be a line formed on a common anatomical structure, such as a particular bone or tissue, and thus the 2D images may coincide or intersect at that line.

[0070] 6 illustrates that multiple 2D or X-ray generated images can be overlaid on an anatomical structure (e.g., in this case, an exemplary skull). However, only one X-ray generated image is shown at a time in FIG. 6. The overlay of two or more 2D images can be defined in part by aligning a first line with a second line 610 using an AR headset. One 2D image shown is overlaid on a test skull, with the user following the line with a needle or trocar 620 simulating a medical procedure.

[0071] 7 shows one 2D image or X-ray generated image displayed as an overlay on an anatomical structure 712. A forward-looking display of multiple detailed views 720 of the X-ray generated image is also displayed. The detailed views may show the X-ray generated image as an overlay using a navigational view (e.g., orthogonal to the anatomy) desired by the medical professional. These are navigational views that show medical device lines and / or graphical markings compared to other X-ray generated image views or other perspectives of viewing the superimposed X-ray views. For example, coronal, axial, and sagittal views may be provided.

[0072] FIG. 8 shows a medical professional simulating the guidance of a needle into an exemplary skull. To acquire images or video, an X-ray interception box can intercept the X-ray-generated video or still images and route the video or still images to the doctor's AR headset rather than transmitting the images to a computer monitor. The AR headset can then have real-time X-ray-generated video, and large monitors in the operating room can be avoided. A still snapshot of the X-ray-generated video can be taken at a point in time, or a still image can be extracted from the video by the AR headset. A QR code with an anatomical structure (e.g., on the side of the skull in FIG. 8) can provide information that allows the AR headset to acquire live video using the IP address of the X-ray interception box obtained from the QR code.

[0073] The physician may perform initial room mapping by looking around the room. The physician can then view each optical tag from several angles to allow the AR headset to match the location of each optical tag. The optical tags may have an image-visible marker that is metal (e.g., titanium or stainless steel) below the optical tag. The image-visible marker may be visible in the X-ray-generated image. The X-ray-generated image may be captured from at least two different angles. The capture of the X-ray-generated image from a particular view is tied to the optical tag that is visible. The AR headset may also provide a virtual monitor on top of the physician's AR headset view that is a live view and changes based on the current input image from the X-ray machine. A foot pedal may be used to start and stop the X-ray exposure.

[0074] In one example, two different X-ray generated images may be acquired at two different or non-parallel angles, but only one X-ray generated image may be shown at a time based on the angle of the physician or observer. For example, the X-ray generated image that is most perpendicular to the observer or physician may be shown. Alternatively, two or more X-ray generated images may be shown simultaneously. The X-ray generated images may be translucent or non-transparent. In one configuration, the physician can switch between two or more different projections provided by the X-ray generated images. The X-ray generated images may be aligned to be coplanar with the anatomical structure of interest, and the images may intersect at any angle between just 1 or 2 degrees and up to 90 degrees or more. The physician or user may pinch the X-ray generated image to move the image along the X-ray projection field through the anatomical structure (e.g., along rays within a virtual cone or virtual pyramid), as discussed in more detail below, to precisely size the X-ray generated image.

[0075] A point or line representing a common anatomical structure (e.g., a common anatomical point or linear structure) may be created by the physician. The X-ray generated images may intersect at the point or line. The physician may guide the needle to a point shown in both X-ray generated images, or the physician may control the needle to follow a line where both X-ray generated images intersect. Thus, there may be a virtual passage through the body that hits a point in the 3D space of the AR headset. A line may be created through two anatomical points in each X-ray generated image, and the planes of the X-ray generated images may intersect along the line. The physician may navigate to a desired point or along a line using only the two X-ray generated images while viewing the anatomical structure from different angles or perspectives (depending on the physician's head position). Thus, the needle may pass through the same two points or proceed along the same line in both X-ray generated images. For example, the needle may move along the same part of a bone in both X-ray generated images.

[0076] Physicians can pass through desired points or lines without a significant amount of advanced planning. The physician simply attaches an optical code with a metal tag to the patient and initiates the X-ray process. This technology enables the use of X-ray-generated images for real-time guidance during the procedure. The physician can take an X-ray and say, "The medical tool looks well positioned from that angle," and then take another X-ray to confirm from a different angle. If the medical tool misses the target, the physician can take another X-ray-generated image. In this method, a limited number of X-ray-generated images (e.g., two or three) are acquired to confirm that the correct anatomical structures are being accessed during the procedure. This contrasts with taking 40+ X-rays as the procedure progresses without using the X-ray-generated images in the AR headset to create a 3D guide.

[0077] The x-ray generated image may not be perpendicular to the x-ray source but at any angle within the x-ray field or pyramid, and the system may extrapolate the x-ray generated image to a desired size. The physician may adjust the size and angle of the x-rays to manually align the x-ray image, if desired. The x-ray beam or x-ray radiation field is generally considered to be cone-shaped, but the x-ray generated image is rectangular due to the shape of the x-ray sensor.

[0078] FIG. 9 shows a second perspective view of a physician using an AR headset to guide a needle into a simulated patient's skull. In one configuration, X-ray video can be used. Additionally, radiopaque instruments can be viewed inside the body in real time on the X-ray video. The AR headset can confirm or align the position and orientation of the C-arm using the emitter (e.g., the top of the pyramid) and detector (e.g., the projection field or base of the pyramid) and an optical code on the person. As the physician drags the X-ray to the center or core of the target anatomy, the X-ray-generated image can be sized to match the anatomy. The end result can be a live X-ray-generated image with accurate scaling of the anatomy superimposed on the patient via the AR headset, and the medical instrument can be visible as the medical procedure is performed. Similar results can be provided by a series of static X-ray-generated images.

[0079] In one configuration, when X-rays are being actively projected, a colored (e.g., red) border, virtual ray, or wavy line may be shown around the video, or a wavy line may be shown coming down from the X-ray emitter to warn of X-ray exposure. Additionally, the AR headset may indicate how much X-ray scatter there is and where it is. These notifications may be shown to the physician in the headset (in real time) when the X-ray machine is currently active (emitting radiation). Additionally, sound may be used as feedback to alert the physician to the active generation of X-ray radiation.

[0080] 10 illustrates a system and method for scaling and aligning X-ray-generated images with a patient's anatomy using an augmented reality (AR) headset. The technology may use an imaging device 1002 that is a smaller, portable C-arm that includes an X-ray detector 1004 plate and an X-ray source 1006. The imaging device 1002 may be on a cart with wheels, and smaller imaging devices may be made for use in imaging legs and arms as well as anatomical extremities. For example, a patient's arm 1008 may be imaged in a portable C-arm in an operating room, and then the patient may be transported to a nearby operating table to perform a medical procedure on the arm 1008.

[0081] The position and orientation of the X-ray device 1002 can be matched to the AR headset. The X-ray device can be detected by using markers, by identifying an optical code 1010 on the X-ray device, or by using pattern recognition to detect the device's shape or outline. The X-ray device can be a small C-arm, a C-arm, a mobile X-ray device, or another semi-mobile X-ray device. For example, one or more markers, optical codes, QR codes, or APRIL codes 1010 can be placed on the small C-arm. The optical code can be used to link the AR headset to a live view of what the X-ray device is capturing; this link can use an image router to which the AR headset connects. For example, live X-ray-generated images or X-ray videos can be oriented in 3D coordinate space based on the position and orientation of the X-ray device. Additionally, live or static X-ray-generated images can also be shown on a nearby flat computer screen.

[0082] Live X-ray generated video can be streamed to the AR headset, and still images of the X-ray video can be captured using voice commands, virtual controls, or physical controls connected to the X-ray device or the AR headset. The user can also crop portions of the captured still X-ray generated image if desired. When the AR headset is used, the X-ray generated image can be displayed on the AR headset's waveguide or display glasses so that it is positioned in the area where the X-ray generated image is taken. This can provide a view that appears to be within the projection field (e.g., X-ray pyramid) of the X-ray radiation area. A QR code 1010 or another optical code can provide location information for the X-ray device so that the AR headset can detect where the X-ray detector is located in 3D space.

[0083] The X-ray generated image can be displayed on an X-ray detection plate or anywhere within the X-ray projection field as long as the X-ray generated image is scaled to that point within the X-ray projection field. An X-ray generated image 1012 captured by an X-ray device of a patient can be superimposed using an AR headset at a location associated with the X-ray projection field as defined in part by the position and orientation of the X-ray device and the geometric attributes of the X-ray projection field. The geometric attributes can include attributes of the X-ray image that allow the X-ray image to be scaled, including the focal length (from the X-ray device source to the detector) and the physical size of the X-ray detector. The image resolution can be used as a geometric attribute and can be calculated from information known about the system. With this information and alignment to the patient's anatomy, the X-ray image can be displayed at any location within the projection field or along a ray from the X-ray source to the center of the X-ray detector when the X-ray was taken of the patient's anatomy. As the X-ray travels from the source to the detector, the X-ray generated image can be enlarged, starting from a small point, until the X-ray generated image is the size of the detector. This forms a "pyramid" or "cone" for the projection or radiation field.

[0084] A user of the AR headset may identify a 3D point of interest in the projection field. The X-ray generated image may then be scaled within the X-ray projection field to include the 3D point of interest. In an alternative configuration, a movement instruction may then be received from the user of the AR headset to move the X-ray generated image to a location that includes the 3D point of interest between the detector array and the X-ray emitter. The movement instruction may be a drag instruction on the X-ray generated image. Alternatively, the movement instruction may be performed using a cursor or graphical buttons displayed on a user interface of the AR headset.

[0085] In one example, a user may move one or more corners of an x-ray generated image. As the corner points are moved, the scale and position of a portion of the x-ray generated image may be modified within the x-ray projection field based in part on the position of the point in the x-ray generated image associated with the geometry of the x-ray projection field. The geometry of the x-ray projection field may be defined using the resolution of the detector array and the distance between the x-ray emitter and the detector array.

[0086] In some applications, the distance between the X-ray emitter and the detector array may not need to be known. If the anatomical target is centered in the X-ray beam, the intersection point can be accurately located without knowing the distance between the X-ray emitter and the detector array. However, if the anatomical target is not centered in the X-ray beam (as is the case in most cases), geometric distortion effects may occur. Knowing the distance between the X-ray emitter and the detector array may allow projection distortion to be taken into account in the depth calculation. Marking or detecting common anatomical structures may also assist in such depth calculations. Unless the anatomical structure is centered, the geometry generally cannot be calculated. However, if the distance from the source to the detector is known, the geometry of off-axis anatomical structures can be determined.

[0087] In another example, a user interface of the AR headset may receive a selection of a corner of the X-ray generated image from a user. Movement instructions for the corner may then be received from the user. The position of the corner of the X-ray generated image may be adjusted relative to the X-ray device (and secondarily the AR headset). A magnification or reduction of a portion of the X-ray generated image may then be set based on the location or distance of each corner of the X-ray generated image relative to the detector array of the X-ray device. The scale of the X-ray generated image may be modified by increasing or decreasing the scaling of the X-ray generated image as the X-ray generated image moves closer to or farther from the X-ray source of the X-ray projection field. The scaling of the X-ray generated image for initial viewing may use at least one of the focal length of the X-ray device, the resolution of the X-ray receiving plate, markers on the human patient's anatomy, calculated geometric boundaries of the X-ray projection field, and / or optical tags on the human patient's anatomy that determine where the patient's anatomy is located within the 3D coordinate system of the AR headset. This means that the x-ray generated image can be appropriately scaled based on known geometric attributes of the imaging device and / or measurable aspects (eg, anatomy) of the patient being imaged.

[0088] The scale and position of the x-ray generated image can be modified to more accurately represent projection distortions that may exist at that measured or geometric point within the x-ray projection field. This can correct for projection contraction or expansion in the x-ray generated image. More specifically, as the x-ray generated image moves away from the x-ray source, the x-ray generated image should be enlarged to match the size of the real object closer to its location. Otherwise, the x-ray generated image, which is a projection of the human anatomy in the x-ray projection field, will be too small (or too large) to match the actual human anatomy. Conversely, as the generated x-rays move toward the x-ray source, image scaling can decrease.

[0089] The reason for image scaling, or image enlargement and reduction, is that X-ray-generated images have projection distortions that can be addressed through reduction or enlargement before displaying the image next to the anatomical structure of interest. For example, an X-ray source can be considered a point source. If an object such as a ball is in the X-ray field, the same ball at different depths from the X-ray source will result in different sized projections. When the ball is close to the X-ray source, the ball will appear larger than its actual size. To adjust for projection distortion, the distance between the X-ray source and the X-ray detector or the focal length plus the resolution of the plate can be used. Anatomical structures close to the X-ray detector or X-ray plate can be displayed on the X-ray-generated image that are close to their true anatomical size. However, if the anatomical structure is at a shallower depth (e.g., higher) within the X-ray projection field, the user can pinch and drag the X-ray-generated image to move it toward the X-ray source, reducing the overall size of the image when viewed through the AR headset.

[0090] For the scaling to work well, a calibration is obtained for the C-arm or other X-ray imaging device being used. That is, the distance between the X-ray detector and the X-ray source can be measured, and the size of the X-ray detection plate can be known. Alternatively, a radiopaque object such as a ball or geometric shape can be used to calibrate the scaling of the X-ray image. In yet another alternative, if sufficient geometric information is known about the X-ray imaging device, the distance between the X-ray source and the X-ray detector can be calculated.

[0091] Another method for calibrating the imaging device can be to use optical markers or optical tags to determine the known geometry of the imaging device. The resolution of the X-ray detector plate may also be known in advance, and the resolution of the X-ray detector plate will generally remain the same. Knowing the geometry of the imaging device and the X-ray detector plate can also allow for the calculation of the focal length. The plate resolution, focal length, and size and shape of the projection field (e.g., radiation pyramid or radiation cone) can all be measured or determined.

[0092] This technique can also be used with portable, handheld X-ray guns where the distance between the X-ray emitter and the X-ray detector is known. In this example, the X-ray-generated image can be linked to the person's anatomy using an optical code. When the portable X-ray gun is removed, the X-ray-generated image can remain as an overlay on the person's anatomy.

[0093] In another example, the arm or leg may be held at an arbitrary distance above the X-ray detector. The X-ray-generated image shown on the AR headset from what the detector plate captures may appear smaller than the actual arm viewed through the AR headset lenses. The user may then pinch and drag on the X-ray-generated image viewed through the AR headset, and the system may then size the X-ray-generated image to the actual or true size of the anatomical structure or arm viewed in the AR headset. If there are two different structures whose images are desired to match, the user may drag one portion of the X-ray-generated image to the size of the matching structure, and then the user may drag a second portion of the X-ray-generated image to a different depth in the X-ray projection field, so that each structure in the X-ray-generated image can be at or close to the actual size of the desired anatomical structure. In this situation, the X-ray-generated image may be at an oblique angle to the X-ray source and X-ray detector. Due to the ability to select and move points on the X-ray-generated image, the user may choose the desired scaling for different anatomical structures.

[0094] In one configuration, the X-ray generated image may scale pixels based on the pixel's depth in the projection field formed by the X-ray projection field. This allows a user to match the X-ray generated image to the actual size of the anatomical structure of interest. Therefore, the X-ray generated image may be scaled differently at different edges. The X-ray generated image, which is a 2D image, may be scaled orthogonal to the X-ray orientation along the pyramid, or the X-ray generated image may be scaled diagonally in the pyramid. This may allow the anatomical size of the X-ray generated image to match the actual anatomical structure when viewed through the AR headset. If a medical professional holds a person's knee on a C-arm and the knee is tilted down, the X-ray generated image may be corrected to match the size of the knee's anatomical structure as the knee tilts in the X-ray projection field.

[0095] Optical tags (e.g., AprilTag) on ​​a person's anatomy, such as an arm or leg, can also be used to determine the size of the generated x-ray. The size of the optical tag is known in advance. For example, the optical tag can be 1 cm or 2 cm square. Identifying the optical tag with an AR headset also identifies where the surface or skin of the body part is located. The depth of the x-ray-generated image can match the surface layer of the anatomy, or the x-ray-generated image can be set to a selected depth near or below the optical tag.

[0096] FIG. 11 shows an example of an X-ray generated image 1012 that has been moved and scaled compared to FIG. 10 to match the anatomical scale of the patient's anatomy at a point closer to the X-ray detector plate 1004 when viewed through an AR headset. In this case, the X-ray generated image 1012 has been enlarged. If the X-ray generated image is moved to a point that appears to be in the center of the anatomy (e.g., the center of the patient's arm), the X-ray generated image can be enlarged, and by scaling the X-ray generated image using the geometric attributes of the X-ray device, the scaling can match the actual size of the arm. Thus, the X-ray generated image can have its position and scale modified to correspond to the position and scale of the patient's anatomy as viewed through the AR headset. This means that the size of the X-ray generated image can be adjusted to be similar to or match the size of the anatomy being imaged. More specifically, the x-ray generated image 1012 may be superimposed at locations such as intersecting a human body part, adjacent to an anatomical structure of interest, proximate to the human skin, aligned with a placement angle of the human patient's anatomy, at the center of mass of the body part in at least one axis, or at a location that bisects the human anatomy in at least one axis.

[0097] In another example, the position and orientation of the X-ray generated image may be set to the position and orientation of the expected path of a medical instrument through a patient's anatomy during a medical procedure. If a needle moves along a particular plane or to a particular point in the patient's actual anatomy, the X-ray generated image may be positioned in the 3D coordinate system of the AR headset so that the medical professional can follow the plane indicated by the X-ray generated image while performing the medical procedure.

[0098] As described above, one or more 2D images may be linked to a portion of a person's anatomy, and as the patient's anatomy moves, the 2D images may move with the patient's anatomy while maintaining their relative position and orientation relative to the patient's anatomy. The X-ray-generated image may move as the patient's anatomy moves while maintaining a reference alignment with the patient's anatomy. To do this, a marker having an image-visible marker may be identified on the person's anatomy from which the X-ray-generated image is acquired. The marker may be an optical code, a visual marker, a colored visual marker, a metallic marker, an IR reflector, an IR marker, or some other type of visible marker. The AR headset may then determine that the anatomical structure is moving and / or outside the X-ray projection field by detecting that the marker is moving. The 2D X-ray-generated image may then be moved and realigned to the moving anatomy by using the image-visible marker in the X-ray-generated image and the marker referenced to the AR headset. Anywhere the patient's anatomy moves (e.g., the patient's arms or legs may move), the 2D x-ray generated images may be moved and maintain their original orientation and position relative to the patient's anatomy as defined by the projections used to capture the 2D x-ray generated images using the medical imaging device. Positional modifications made to the 2D x-ray generated images by the user relative to the patient's body may also be maintained.

[0099] As described above, after the patient removes the user's arm from the X-ray machine, the user can still view the X-ray-generated image away from the X-ray machine while using the AR headset. At this point, corrections can still be made to the positioning and orientation of the 2D X-ray-generated image. Even when the patient's anatomical body part (such as the arm) is away from the X-ray machine, the user can drag or adjust the depth of the X-ray-generated image, so that the user can drag the 2D X-ray-generated image through the arm to any depth when the patient is on the operating table. This is performed by creating a virtual C-arm or virtual X-ray space away from the X-ray machine, which can be used during the medical procedure. Whenever the 2D X-ray-generated image is moved, the virtual X-ray space can move along with the X-ray-generated image.

[0100] Each of the multiple 2D X-ray generated images captured using the X-ray device may be a non-parallel projection or a non-parallel spatial arrangement. Each of the multiple X-ray generated images may be positioned relative to one another based on an initial spatial arrangement of each X-ray generated image relative to the patient's anatomy. Each of the X-ray generated images may be linked to a virtual marker on a common patient's anatomy in the X-ray generated images, an optical code on the patient's anatomy, a morphometric model of the human anatomy, or a virtual model of the human anatomy.

[0101] One useful result of this technology is that it is possible to use 2D X-ray-generated images organized as 3D spatial mapping or 3D navigational references, avoiding additional, more expensive imaging such as CT scans or MRIs before a medical procedure is performed. As a result, this technology uses widely available 2D X-ray-generated images to provide an augmented reality (AR) view. Similarly, any optical and image-visible tags that need to be used can be attached to a person's body immediately before surgery. This means that additional, non-parallel imaging sessions may not be required, and only one imaging session may be provided before the medical procedure.

[0102] Furthermore, the 2D X-ray generated images and X-ray pyramids can be integrated with any other imaging modalities that may be available. Examples of other imaging modalities that may be integrated with the present technology may be MRI (Magnetic Resonance Imaging) modality, CT (Computed Tomography) scanning modality, Positron Emission Tomography (PET) modality, ultrasound modality, fluorescence modality, Infrared Thermography (IRT) modality, 3D mammography, or Single Photon Emission Computed Tomography (SPECT) scanning modality, etc. Image datasets from another image modality may be registered to the person's body using optical codes, image visible markers, and / or imaging of image visible markers visible in the X-ray generated images.

[0103] 12A shows that a first 2D X-ray generated image 1214 may be a projection through an anatomical structure 1212, such as a person's arm. The pyramid in FIG. 12A represents the spread or projection field of X-ray radiation from the X-ray source. The zoom of the first 2D X-ray generated image 1214 may be set to bisect the anatomical structure 1212.

[0104] FIG. 12B shows that a second 2D image 1220 of the anatomical structure 1212 can be captured. The second 2D X-ray image 1220 in a second orientation can be linked to the position and orientation of the person's arm using an optical tag or the shape of the arm. The projection fields or cones in FIGS. 12A and 12B can be set as virtual cones for each projection. Furthermore, the X-ray generated image views can be displayed separately based on which direction the arm is facing. Thus, the X-ray generated image can be turned on only when it is the image closest to the medical professional. Through the AR headset, the user can "examine" the arm at several angles, and each X-ray generated image may appear to be the true size of the patient's anatomy. Because the X-ray generated images are associated with the optical code, the X-ray generated images can appear as if they were attached to the arm and aligned with the correct anatomy as the arm moves.

[0105] As previously mentioned, markers or optical codes may link the image and virtual cone or pyramid to the person being imaged. One or more of the virtual pyramid and the X-ray generated image may be moved away from the operating table. The X-ray generated image may then maintain its originally captured position, orientation, and scaling relative to the anatomy.

[0106] 12C shows a first x-ray generated image 1214 and a second x-ray generated image 1220 overlaid together on the anatomy 1212. Dotted lines 1230 indicate that additional 2D x-ray generated images may be captured at any angle and overlaid on the anatomy 212, thus creating an additional guide layer for 3D spatial mapping.

[0107] In an example medical procedure, a medical professional may be operating on a patient's femur. Due to a fracture, the physician may want to place a rod into the femur at a certain level within the bone. It may also be important for the physician to overlook certain items that may cause complications in the procedure. Therefore, a projection field (e.g., any geometrically shaped radiation field) may be established, and an instrument tracker may be attached to a needle or screw. The anatomical region of interest is the path of the screw or needle. As a result, the x-ray-generated image may be shown in the plane of interest or the desired path of the medical device. The physician can see the required depth and trajectory and then align the tool with those trajectories and depths. Furthermore, the physician may be able to see the exact relative size of the anatomical structures while viewing the 3D guidance. Furthermore, as the physician moves the tool up and down, the x-ray-generated image may be referenced to provide the physician with the exact relative size of the anatomical structure where the tool is currently located.

[0108] FIG. 13A illustrates how a first object 1302 may appear smaller in a 2D x-ray image when the image is reduced to match the size of a second object 1304. However, FIG. 13B illustrates that a corner 1310 can be moved downward within the x-ray projection field or cone, magnifying the portion of the image capturing the first object 1302. Thus, each object can be viewed closer 1320 or at their actual size in the image. While FIG. 13B illustrates moving a single corner, each corner of the image or any point on the image can be selected for movement. Continuity between that point and other points can be maintained relative to the 2D plane by interpolating the point being moved and the amount of movement applied to the other points. This can result in the 2D plane of the 2D x-ray-generated image being doubly oblique or having other non-orthogonal orientations. For example, if an anatomical structure of interest to a medical professional is not parallel to the x-ray detector plate, the plane of the 2D x-ray-generated image can be aligned with the orientation of the anatomical structure of interest.

[0109] This technology provides the ability to navigate a patient's anatomy with medical or other tools in 3D using two or more 2D X-ray generated images, which can be displayed in an AR headset as intersecting planes that provide a 3D navigation reference.

[0110] When 2D X-ray-generated images intersect using a line defined by a common anatomical point (e.g., an orthogonal intersection view), the medical professional can navigate the instrument along the virtual line. The AR headset's graphical interface can use color to indicate whether the medical instrument is aligned with the virtual guide line. For example, a blue line can be shown emanating from a real or virtual part of the instrument. Then, when the medical instrument is aligned with the yellow virtual guide line, a green indicator can be displayed if the physician is aligned. Any other color scheme can also be shown for accurate alignment. The instrument guidance interface can also show the physician where the medical tool is aligned on detailed views at the bottom of the AR headset interface. These are navigation views, such as a coronal view and a sagittal view. On the detailed views in the 3D coordinate system, there can be a virtual line created by the physician, which can be used as a needle guide line. In these navigation views, the physician can also see depth, which may be difficult to see. The physician can see whether the instrument is aligned with the virtual tracker. This guidance for the physician encourages the physician to move the instrument accurately.

[0111] In the past, doctors without guidance would use medical tools, such as needles, on a patient's body, but the medical tool was not always near the target the medical professional wanted to hit. Therefore, doctors could align the medical tool with the desired anatomical structure through trial and error. This generally resulted in the medical tool or the tool tip going off course or missing the target. However, the accuracy of medical procedures can be dramatically improved by using an AR headset that uses two or more X-ray-generated images to find points or virtual lines as a 3D path to guide the doctor. This technology allows the doctor to identify points or plan a path independently in each plane. Therefore, this technology can be a very accurate guide to the points or lines used by medical professionals.

[0112] In another example, two x-ray images of a femur may be acquired. The target of a medical procedure may be the bone marrow cavity of a patient's femur. A physician may mark two sites on the first x-ray image representing the bone marrow cavity of the femur, resulting in a virtual line. The markings may be independently repeated on the second x-ray image to provide a second virtual line on a common anatomical structure. These two lines may intersect, providing an intersection point on both x-ray images. This line may define a virtual navigation path through a 3D coordinate space viewed by the VR headset. The 3D spatial mapping is defined using the two 2D x-rays, and the physician can view the defined trajectory in 3D coordinates in the AR headset.

[0113] In another example, shown in FIG. 13C, the use of one line and one point on two x-ray-generated images can be used for pedicle placement. From the first view on the right, the physician sees a channel 1350 in the end of the screw to allow the screw to be driven into the vertebra and marks the channel with a first point. From the second view, orthogonal to the top view, the physician will see where the shaft of the pedicle screw is desired to be placed. The physician can draw a line 1360 on the lateral view as to where the screw should be driven into the vertebra. From the first view, only one point is required. A virtual line can be generated between the x-ray source and the first point. This means that only three points in space can be marked for navigation purposes. Two points can be explicitly defined on the x-ray-generated image of the second view, one point can be defined on the x-ray-generated image of the first view, and another point can be implicitly defined from the x-ray source. If the first view is perpendicular or orthogonal to the second X-ray-generated image, a line can be formed from the X-ray source to the marked point. These lines can be aligned to define the path of the pedicle screw. Because the third point does not provide a specific depth in space, the physician can visually define the depth on the line created using the third point and the X-ray source. This configuration avoids the need for a pre-operative CT scan, since the physician has identified a virtual target in this process. If the points define a line to the X-ray source and the second image has a line, the physician can set the depth of a medical device (e.g., a screw, probe, needle, etc.) on the virtual line created using the points.

[0114] 14 is a flowchart illustrating an exemplary method for creating a 3D spatial mapping or guide. One operation in the method may be receiving 2D image data of a patient's anatomy from a medical imaging device, as in block 1410. In one example, the 2D image data is an ultrasound image.

[0115] Another operation may be to identify the location of the 2D image data in the 3D coordinate system of the augmented reality (AR) headset, as in block 1420. The location of the imaging device may be identified by matching a marker or optical code on the imaging device (e.g., an ultrasound imager). Alternatively, the shape of the ultrasound imager may be identified, which may identify the placement (i.e., position and orientation) of the ultrasound imager.

[0116] A graphical mark may be received from a user for a point in the 2D image data, as in block 1430. The graphical mark may have a depth associated with the point as reported in the ultrasound image or by the ultrasound transducer.

[0117] As in block 1440, 3D coordinates of points in a 3D coordinate system can be calculated using the location of the medical imaging device relative to the AR headset to generate 3D spatial mapping and target human structures. 2D images acquired from the medical imaging device can be displayed through the augmented reality (AR) headset while viewing the patient's anatomical structures. Additionally, treatment guidance indicators (i.e., virtual lines or arrows) from medical tools identified by the AR headset can be displayed at the 3D coordinates of the points.

[0118] 15 illustrates an example of a method for 3D spatial mapping. One operation in the method may be receiving a plurality of 2D images of a patient's anatomy representing non-parallel projections from a medical imaging device, as in block 1510. The 3D images may be at least one of X-ray generated images, ultrasound images, CT generated images, MRI generated images, or other 2D medical images.

[0119] When capturing the multiple 2D images, a position and orientation of the imaging device in a 3D coordinate system may be identified with respect to the patient and the augmented reality (AR) headset, as in block 1520. At least two graphical marks may be received from a user for a common anatomical structure in the multiple 2D images, as in block 1530. The graphical marks may be at least one of a point, a line, a navigation path, or a 3D target.

[0120] The 3D coordinates of the common anatomical structure within the 3D coordinate system for the patient may be determined within the 3D coordinate system by taking the intersection of projection lines from the x-ray source to at least two graphical marks in the plurality of 2D images, as in block 1540. The 2D images may be registered by using a virtual line placed on the common anatomical structure in each of the 2D images.

[0121] 2D images from a medical imaging device can be displayed using an augmented reality (AR) headset to view the patient's anatomy to generate 3D spatial mapping. Navigation paths from medical tools identified by the AR headset to the 3D coordinates of common anatomical structures can also be provided.

[0122] 16 illustrates a method for aligning an X-ray generated image with a person's anatomy using an augmented reality (AR) headset. One operation in the method is to identify markers on the person's anatomy using sensors in the AR headset, as in block 1610. The position and orientation of the X-ray device in a 3D coordinate system can be aligned with the AR headset, as in block 1620.

[0123] The 2D x-ray generated images captured by the x-ray device may be registered with the human anatomy, as in block 1630, using at least one of the markers, the position and orientation of the x-ray device, or geometric attributes of the x-ray projection field.

[0124] A further operation of the method may be, as in block 1640, maintaining alignment of the 2D X-ray generated image with the human anatomical structure using markers on the human anatomical structure when the anatomical structure moves outside the X-ray projection field of the X-ray device. For example, the anatomical structure may move outside the X-ray projection field with the markers. Thus, the X-ray generated image may be re-aligned with the human anatomical structure using the markers on the human anatomical structure. The markers may enable detection of the orientation of the human anatomical structure in the 3D coordinate system of the AR headset. The X-ray generated image may be aligned to the orientation of the anatomical structure using the AR headset by using the visible markers in the X-ray generated image and the orientation of the anatomical structure.

[0125] Some of the functional units described herein may be labeled as modules to more specifically emphasize their implementation independence. For example, a module may be implemented as a hardware circuit comprising custom VLSI circuits or gate arrays, off-the-shelf semiconductors such as logic chips, transistors, or other discrete components. A module may also be implemented in a programmable hardware device such as a field programmable gate array, programmable array logic, programmable logic device, or the like.

[0126] Modules may also be implemented in software for execution by various types of processors. An identified module of executable code may comprise one or more blocks of computer instructions, which may be organized as, for example, an object, a procedure, or a function. Nevertheless, the executable files of an identified module need not be physically located together, but may comprise heterogeneous instructions stored in different locations that make up the module and that, when logically combined together, achieve the purpose stated for the module.

[0127] Indeed, a module of executable code may be a single instruction, or many instructions, and may even be distributed across several different code segments, among different programs, and across several memory devices. Similarly, operational data may be identified and illustrated herein within modules, and may be embodied in any suitable form and organized within any suitable type of data structure. Operational data may be collected as a single data set or distributed across different locations, including across different storage devices. Modules may be passive or active, including agents operable to perform a desired function.

[0128] The techniques described herein may be stored on computer-readable storage media, including volatile and nonvolatile, removable and non-removable media implemented in any technology for storage of information such as computer-readable instructions, data structures, program modules, or other data, including, but not limited to, RAM, ROM, EEPROM, flash memory or other memory technology, CD-ROM, digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other computer storage medium that can be used to store the desired information and the techniques described.

[0129] Devices described herein may also include communications or networking devices and connections that allow devices to communicate with other devices. Communications connections are one example of communications media. Communications media typically embodied computer-readable instructions, data structures, program modules, and other data in a modulated data signal, such as a carrier wave or other transport mechanism, and include any information delivery media. A "modulated data signal" means a signal that has one or more of its characteristics set or changed in such a manner as to encode information in the signal. By way of example, and not limitation, communications media include wired media, such as a wired network or direct-wired connection, and wireless media, such as acoustic, radio frequency, infrared, and other wireless media. As used herein, the term computer-readable media includes communications media.

[0130] Furthermore, the described features, structures, or characteristics may be combined in any suitable manner in one or more embodiments. In the preceding description, numerous specific details, such as embodiments of various configurations, are provided to provide a thorough understanding of embodiments of the described technology. However, those skilled in the art will recognize that the technology can be practiced without one or more of the specific details, or with other methods, components, devices, etc. In other instances, well-known structures or operations have not been shown or described in detail to avoid obscuring aspects of the technology.

[0131] Although the present subject matter has been described in language specific to structural features and / or operations, it should be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features and operations described above. Rather, the specific features and operations described above are disclosed as exemplary forms of implementing the claims. Numerous modifications and alternative arrangements can be devised without departing from the spirit and scope of the described technology.

Claims

1. receiving 2D image data of a patient's anatomy from a medical imaging device; Identifying a location of the 2D image data in a 3D coordinate system of an augmented reality (AR) headset; receiving a graphical mark from a user for a point within the 2D image data; and calculating 3D coordinates of the points in the 3D coordinate system using a location of a medical imaging device relative to the AR headset to generate 3D spatial mapping and target internal structures of the person.

2. 10. The method of claim 1, further comprising displaying the 2D image data while viewing the patient's anatomy through an augmented reality (AR) headset.

3. The method of claim 1 , further comprising displaying a treatment guidance indicator for a medical tool or an augmented tag identified by the AR headset at the 3D coordinate of the point.

4. The method of claim 1 , wherein the 2D image data is an ultrasound image.

5. The method of claim 1 , wherein the medical imaging device has optical markers on a surface of the medical imaging device to identify the placement of the medical imaging device and to provide 3D coordinates of the 2D image data.

6. receiving a plurality of 2D images of a patient's anatomy representing non-parallel projections from a medical imaging device; identifying a position and orientation of an imaging device in a 3D coordinate system relative to the patient and an augmented reality (AR) headset when capturing the plurality of 2D images; receiving at least two graphical marks from a user for a common anatomical structure in the plurality of 2D images; and determining 3D coordinates of the common anatomical structure in the 3D coordinate system for the patient by taking intersections of projection lines from an x-ray source to the at least two graphical marks in the plurality of 2D images.

7. 10. The method of claim 6, further comprising displaying 2D images from the medical imaging device while viewing the patient's anatomy using an augmented reality (AR) headset to generate 3D spatial mapping.

8. The method of claim 6 , wherein the graphical mark is at least one of a point target, a line target, a navigation path target, a 2D area target, or a 3D area target.

9. The method of claim 6 , further comprising providing a navigation path from a medical tool identified by the AR headset to the 3D coordinates of the common anatomical structure.

10. The method of claim 6 , wherein the 2D images are registered by using a virtual line placed on a common anatomical structure in each of the 2D images.

11. The method of claim 6 , wherein the 2D image is at least one of an X-ray generated image, an ultrasound image, a CT generated image, or an MRI generated image.

12. 1. A method for aligning x-ray generated images in 3D coordinate space while viewing a patient using an augmented reality (AR) headset, comprising: receiving a plurality of x-ray generated images of a patient's anatomy; registering the plurality of x-ray generated images using optically visible markers on the patient and associated image visible markers, wherein the associated image visible markers are represented in the x-ray generated images; receiving, from a user, first point markups for annotating anatomical structures on a first x-ray generated image and second point markups in a second x-ray generated image, the first point markups and the second point markups representing a common structure or a medical procedure trajectory of the patient; and aligning the first point markup and the second point markup to generate a 3D spatial mapping using the first X-ray generated image and the second X-ray generated image viewable using the AR headset.

13. The method of claim 12, wherein the optically visible marker is an optical code, a visible marker, a linear barcode, a 2D barcode, a QR code, or an APRIL code.

14. 13. The method of claim 12, further comprising using the AR headset to display the plurality of x-ray generated images superimposed on the patient's anatomy as defined in part by alignment of the first point markup and the second point markup.

15. 13. The method of claim 12, wherein the plurality of x-ray generated images are orthogonal to one another and further include additional points on the first x-ray generated image that form a line with the first point markup, the additional points being aligned with the line.

16. The method of claim 12 , further comprising using geometric attributes of an x-ray projection field to define a position and orientation of the x-ray generated image relative to the patient's anatomy.

17. 13. The method of claim 12, further comprising: moving the x-ray generated image to maintain alignment with the optically visible markers and the associated image visible markers on the patient's anatomy as the patient's anatomy moves relative to an x-ray machine or as the x-ray machine moves relative to the patient.

18. 1. A method for aligning an X-ray generated image to a patient's anatomy using an augmented reality (AR) headset, comprising: Aligning the position and orientation of the X-ray device in 3D coordinate space; using the AR headset to superimpose an x-ray generated image captured by the x-ray device at the position and orientation associated with the x-ray projection field, as defined in part by the position and orientation of the x-ray device and geometric attributes of the x-ray projection field; receiving an identification of a 3D point of interest in the X-ray projection field from a user of the AR headset; and positioning and scaling the x-ray generated image within an x-ray projection field to include the 3D point of interest.

19. 20. The method of claim 18, wherein the X-ray generated image is scaled to match an anatomical scale of the patient's anatomy at the 3D point of interest when viewed through the AR headset.

20. 20. The method of claim 18, further comprising modifying a position and scale of the x-ray generated image to correspond to a position and scale of the patient's anatomy when viewed through the AR headset.

21. 21. The method of claim 20, wherein correcting the position and scale of the x-ray generated image to correspond to the position and scale of the patient's anatomy further comprises superimposing the x-ray generated image at a location that is at least one of: intersecting a body part of a person, adjacent to an anatomy of interest, proximate to the skin of the person, aligned with an angle of alignment of the patient's anatomy of the person, at a center of mass of the body part in one axis, or bisecting the anatomy of the person in one axis.

22. 20. The method of claim 18, wherein the position and orientation of the x-ray generated image are set to the position and orientation of an expected path of a medical instrument through the patient's anatomy during a medical procedure.

23. receiving a selection of at least one point within the x-ray generated image from a user; 20. The method of claim 18, further comprising receiving a drag instruction on the x-ray generated image to move the x-ray generated image.

24. 20. The method of claim 18, further comprising defining the geometry of the x-ray projection field using a resolution of a detector array and a distance between an x-ray emitter and the detector array.

25. identifying a marker using an image visible marker on the anatomy of the person from which the x-ray generated image is obtained; 20. The method of claim 18, further comprising: using the AR headset to register the X-ray generated image to the anatomical structure by using the markers to reference image visible markers in the X-ray generated image.

26. 26. The method of claim 25, wherein the marker is an optical code, an infrared reflector, or a visible marker.

27. capturing a plurality of x-ray generated images from the x-ray device, wherein the patient's anatomy is in a non-parallel projection during each capture of the x-ray generated images; 20. The method of claim 18, further comprising: registering the plurality of x-ray generated images relative to one another based on an initial projection of each x-ray generated image onto the patient's anatomy.

28. 20. The method of claim 18, wherein the x-ray generated image moves while maintaining a projection position and orientation relative to the patient anatomy as the patient anatomy moves.

29. 20. The method of claim 18, wherein the x-ray generated image is linked to at least one of a marker on the patient's anatomy, an optical code on the patient's anatomy, a morphometric model of a human anatomy, or a virtual model of the human anatomy.

30. receiving from the user an identification of a corner 3D point of the x-ray generated image; adjusting the position of the corner of the x-ray generated image relative to the x-ray device to include the 3D point; 20. The method of claim 18, further comprising: setting a magnification or reduction of a portion of the x-ray generated image based on a location of the corner of the x-ray generated image relative to a detector array of the x-ray device.

31. 20. The method of claim 18, wherein scaling the x-ray generated image further comprises increasing or decreasing magnification of the x-ray generated image as the x-ray generated image moves toward or away from an x-ray source of the x-ray projection field.

32. 20. The method of claim 18, further comprising scaling the X-ray generated image for initial viewing using at least one of a focal length of the X-ray device, a resolution of an X-ray receiving plate, a marker on the patient's anatomy of the person, a calculated geometric boundary of the X-ray projection field, or an optical tag on the patient's anatomy of the person that determines where the patient's anatomy is located within a 3D coordinate system of the AR headset.

33. 20. The method of claim 18, wherein the X-ray device is a mini C-arm, a C-arm, a mobile X-ray device, an angiogram machine, or a fixed X-ray device.

34. 20. The method of claim 18, wherein aligning a position and orientation of an X-ray device using the AR headset further comprises aligning the position and orientation of the X-ray device using a marker or optical code on the X-ray device.

35. 20. The method of claim 18, further comprising using an outline of an x-ray device to identify the location and orientation of the x-ray device.

36. 1. A method for aligning X-ray generated images with human anatomy using an augmented reality (AR) headset, comprising: identifying markers on the anatomy of the person using sensors of the AR headset; Aligning the position and orientation of the X-ray device in a 3D coordinate system; registering a 2D x-ray generated image captured by the x-ray device with the anatomical structure of the person based in part on at least one of the markers, the position and orientation of the x-ray device, or geometric attributes of an x-ray projection field; and aligning the 2D x-ray generated image with the anatomical structure of the person using the markers on the anatomical structure of the person to maintain alignment outside the x-ray projection field of the x-ray machine.

37. Maintaining alignment is using the marker to detect when the anatomical structure moves outside the X-ray radiation field; 37. The method of claim 36, further comprising re-registering the x-ray generated image with the anatomy of the person using the markers on the anatomy of the person and image visible markers to maintain an orientation of the x-ray generated image relative to the anatomy.

38. 37. The method of claim 36, further comprising scaling the x-ray generated image by using geometric attributes of a projection field stored with the x-ray generated image when the x-ray generated image leaves the x-ray device.

Images