Image Dataset Registration for AR Headsets Using Anatomical Structure and Data Fitting

Image-visible markers with optical codes enable precise alignment of AR systems with patient anatomy, improving medical procedure accuracy and reducing radiation and dye use.

JP2024543925APending Publication Date: 2024-11-26NOVARAD CORP
View PDF 5 Cites 0 Cited by

Patent Information

Application Number
JP2024531480
Authority / Receiving Office
JP · JP
Patent Type
Applications
Current Assignee / Owner
Priority Date
2021-11-27
Filing Date
2022-11-26
Publication Date
2024-11-26

AI Technical Summary

Technical Problem

Existing augmented reality (AR) systems face challenges in accurately aligning virtual objects with the real-world environment, particularly in medical applications, where high precision is required but current methods are time-consuming and inaccurate.

Method used

The use of image-visible markers with optical codes, such as radiopaque markers and optical codes, to align X-ray generated images with a patient's body in a 3D coordinate system, enabling precise registration of image datasets with the patient's anatomy using augmented reality headsets.

Benefits of technology

This method allows for precise alignment of virtual medical imagery with the patient's body, enhancing the accuracy of medical procedures by providing detailed anatomical views and reducing radiation exposure and dye usage.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure 2024543925000001_ABST
    Figure 2024543925000001_ABST
Patent Text Reader

Abstract

Techniques are described for registering an image dataset with a patient using an augmented reality (AR) headset. The method may include acquiring an image dataset representing an anatomy of the patient. A two-dimensional (2D) radiographically generated image of at least a portion of the patient's anatomy in the image dataset and visible markers may be obtained. The image dataset may be registered to the radiographically generated image by using data fitting. The positions of the visible markers may be defined in the image dataset using the registration with the radiographically generated image. The image dataset may be registered with the patient's body using visible markers in the image dataset relative to the visible markers seen on the patient through the AR headset.
Need to check novelty before this filing date? Find Prior Art

Description

[Technical field]

[0001] Mixed reality or augmented reality is an area of ​​computing technology in which images from the physical and virtual computing worlds can be combined into a mixed reality world. In mixed reality, people, places, and objects from the physical and virtual worlds become blended environments. Mixed reality experiences can be delivered through the use of VR (Virtual Reality) or AR (Augmented Reality) headsets as well as through existing commercial or custom software.

[0002] Augmented reality (AR) is an example of mixed reality in which a live direct (or indirect) view of a physical, real-world environment is augmented or supplemented by computer-generated sensory input such as audio, video, computer graphics, or other data. Augmentation is performed in such a way that real-world locations are seen and associated with environmental elements. With the aid of AR sensor technologies (e.g., the addition of computer vision and object recognition), information about the real world around the user becomes interactive and may be digitally modified.

[0003] A problem faced by AR systems or AR headsets is identifying the position and orientation of an object with high accuracy. Similarly, aligning the position of a virtual element or object with a live view of the real-world environment can be difficult. The alignment resolution of an AR headset may align a virtual object with the physical object being viewed, but the alignment resolution may only be aligned to within a few centimeters. While providing alignment within a few centimeters may be useful for entertainment and less demanding applications, increasing the positioning and alignment resolution of an AR system may be desirable in the fields of science, engineering, and medicine. As a result, the positioning and alignment process may be performed manually, which may be time-consuming, cumbersome, and inaccurate. [Brief description of the drawings]

[0004] [Figure 1A]An example of a technique for registering an image dataset with a human body using X-ray generated images with image visible markers and registration of anatomical structures is presented. [Figure 1B] 1 illustrates an exemplary augmented reality (AR) environment in which an image dataset of a patient can be aligned to an actual view of the patient using one or more image visible markers or an image visible marker with an optical code attached to the patient. [Diagram 2] 1 shows an example of an image visible marker or an image visible marker having an optical code that is attached to the patient's body. [Diagram 3] An example of an X-ray generated image is shown that is registered with an image dataset using data fitting, and the image dataset is then registered with the patient's body using image visible markers or image visible markers with optical codes. [Figure 4A] 1 shows an example of a diagram of a human body with bones and radiopaque markers with optical codes. [Figure 4B] An example of a diagram of a human body as in FIG. 4A is shown with a mapping ray from an X-ray source to the skin of the body to identify the location of an image visible marker on the skin or an image visible marker with an optical code. [Diagram 5] An example is shown of the use of an augmented reality (AR) display to enable the display of X-ray generated images and image projections from image datasets onto a person's body using image visible markers or image visible markers with optical codes. [Figure 6] An example of a fluoroscope is shown that is movable relative to a person's body, and the x-ray generated images and image projections can change position and / or orientation based on the movement of the moving fluoroscope. [Figure 7] A mobile fluoroscope generating X-ray generated images is shown, with the image projection illustrated in a coronal view to allow combined viewing through an augmented reality (AR) headset or AR display. [Figure 8A] A mobile fluoroscope is shown generating X-ray generated images, with the image projection shown in a sagittal view to allow combined viewing through an augmented reality (AR) headset. [Figure 8B] A mobile fluoroscope is shown generating X-ray generated images, and image projection is shown to enable combined viewing through an augmented reality (AR) headset. [Figure 9A] 1 is a flowchart of an exemplary method for registering an X-ray generated image to an image dataset and then registering the image dataset to a patient's body using an augmented reality headset. [Figure 9B] 1 is a flowchart of an example method of registering an X-ray generated image to an image dataset using an augmented reality headset and then registering the image dataset to a patient's body using a radiopaque marker having an optical code. [Figure 10] 1 illustrates an exemplary computing system that can be used to register an image dataset to a human body. [Figure 11] FIG. 1 is a block diagram illustrating an example of a computing system for processing the present techniques. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0005] Techniques are provided that allow one or more visible markers to be identified in an X-ray generated image of a patient using an augmented reality (AR) headset and used to register an image dataset with a person's body in the 3D coordinate system of the AR headset during a medical procedure. The visible markers may have two distinct aspects: 1) the visible markers may be visible in the X-ray generated image, and 2) the visible markers may be optically visible to the optical camera of the AR headset and registerable therewith (or using infrared). The image visible markers are an example of an aspect of a visible marker that can be seen in a non-optical imaging modality. In one configuration, the image visible markers may each include an optical code (e.g., a radiopaque material, metal, or titanium shape having an optical code) formed therein. In another configuration, the image visible marker may be a shaped radiopaque material (e.g., a radiopaque pattern, a metal ball, a radiopaque cross, etc.), but the image visible marker may not have an optical code formed therein (e.g., the image visible marker is a blank shape without a code or letter). FIG. 1 shows that a projection image data set 150 may be created using an image data set 148 captured by a medical imaging device. The projection image data set 150 may be a maximum intensity projection (MIP). In addition, the projection image data set 150 may first be aligned with an X-ray generated image 152 using shared anatomical structures and data fitting, and the X-ray generated image 152 may include one or more image visible markers 154 or image visible markers with optical codes. The image data set 148 may receive the image visible markers copied from the X-ray generated image 152. Alternatively, the image dataset 148 may be fused with the x-ray generated image 152, and then the image visible marker locations are known in both the x-ray generated image 152 and the image dataset 148. In yet another option, the image visible marker locations from the x-ray generated image 152 may be correlated or referenced to the image dataset 148 by simply calculating and storing the locations of the image visible markers using the visible markers or optical codes seen in the x-ray generated image and on the patient.The image dataset 148 may then be registered with the body of the person 156 using one or more image visible markers 154 or image visible markers 154 having optical codes on the person's body.

[0006] The image dataset 148 may be a previously acquired image of a portion of the person's body using a non-optical imaging modality (e.g., using MRI (magnetic resonance imaging), CT (computed tomography) scan, x-ray, etc.). The image dataset 148 may be registered to the person's body 156 using image visible markers 154 or image visible markers having an optical code placed on the person's body. In one exemplary configuration, the image visible markers 154 may have an optical code formed in or embossed on the image visible marker. More specifically, a titanium shape (e.g., a square) may have an optical code etched, embossed, or cast into the metal of the titanium shape. In another example, both the image visible markers 154 and the optical code (e.g., AprilTag or 2D optical barcode) may be formed in a non-metallic radiopaque material to facilitate registration of the image dataset 148 with the person's body 156. Thus, the registration relationship between the image dataset 148 and the person's body 156 can be indirectly determined through the image visible markers that are visible on the person's body through the AR headset and the x-ray generated image 152 having the image visible markers. This registration can be done even if the image dataset 148 does not initially have the image visible markers in the image dataset.

[0007] Image visible markers are markers that can be seen in non-visible imaging modalities, such as captured radiology images or image datasets, which may or may not be optically visible to the AR headset. The image dataset may capture structures of the human body using a non-optical imaging modality. The image visible markers in the image dataset may be used to align the image dataset with image visible markers on the patient's body that are visible through the AR headset, or image visible markers with optical codes. The image visible markers in the image dataset may be obtained from an X-ray generated image with blank image visible markers or image visible markers with optical codes formed in radiopaque markers. In some examples, the image visible markers may be radiopaque markers or MRI beads, etc. The optical codes that may be optionally formed in the image visible markers may enhance the recognition and alignment capabilities of image visible markers detected on the patient's body using the AR headset, and enhance the ability to detect the location of the image visible markers in the X-ray and / or image datasets.

[0008] In another configuration of the present technology, an augmented reality (AR) headset or AR display can display X-ray generated images and image projections from an image dataset in registration with respect to the person's body. The position and orientation of the X-ray generated images as image projections may be defined by the position and orientation of a movable fluoroscope with respect to the person's body. The mobile imaging device may allow the medical professional to change the orientation of the imaging device or move the emitter, detector, transducer, and / or imaging components of the imaging device with respect to the person's body. One or more optical codes on the fluoroscope can be used to determine the position and orientation of the fluoroscope with respect to the person's or patient's body. The X-ray generated images and image projections are registered using the shared anatomical structures and data matching (as described above and later in this specification) and displayed through the AR headset, and a portion of the person's body may be viewed and fluoroscopically imaged through the AR headset. One or more image visible markers on the person's body, or image visible markers with optical codes and one or more optical codes on the fluoroscope can be used. Image visible markers or image visible markers having optical codes on the person's body and optical codes on the fluoroscopy device can be identified in data (e.g., visual data and IR data) captured by sensors in the AR headset.

[0009] In one exemplary configuration, the image visible marker with an optical code on the person's body can have an optical code formed on the image visible marker. For example, the optical code can be combined or matched with the image visible marker. Furthermore, the optical code can be engraved or cut into the image visible marker that is a metal or radiopaque material. This allows the image visible marker with an optical code on the person's body to be used to align an image data set (e.g., a radiological image) to the person's body when viewed through an AR display (such as an AR headset).

[0010] Image projections may be created from the image data set based on the position and orientation of the fluoroscope. The X-ray generated images and / or image projections from the fluoroscope may be aligned with the person's body based on image visible markers (e.g., radiopaque objects), or image visible markers with optical codes, and / or one or more optical codes defining the position and orientation of the fluoroscope. Additionally, the X-ray generated images may be virtually displayed in the AR headset at a position relative to the person's body where the X-ray beam is passing through the person's body, and the X-ray generated images may be aligned to overlay a portion of the person's body being imaged with the X-ray beam. The image projections may be oriented parallel to the X-ray generated images (i.e., perpendicular to the X-ray beam) and may be displayed in the AR headset as being virtually in at least a portion of the path of the X-ray beam. The aligned images may be displayed using the AR headset along with a real-world or real view of the patient, or the aligned images may be displayed on a separate AR display (e.g., a separate display screen). This process combines, aligns, and orients live X-ray generated images, image datasets (e.g., augmented reality images or image projections), and the actual view of the person, thereby allowing useful aspects of the X-ray generated images (e.g., navigation of radiopaque objects within the person's body) and the image datasets (e.g., better tissue contrast, etc.) to be combined during a medical procedure.

[0011] In another configuration using X-ray generated images, one or more optical codes of the fluoroscope can be used to detect changes in the position and orientation of the fluoroscope relative to the person's body. The image projection and the X-ray generated image position and orientation may then be modified as defined by the changes in the position and orientation of the fluoroscope. For example, the movement of the projection of the image data set may be co-localized and / or synchronized to match the X-ray generated image based on the changes in the orientation and position of the fluoroscope. The zoom of the fluoroscope may also be detected using a radiopaque object on the person's body, and the size of the radiopaque object may be used to adjust the size of the image projection when viewed on the AR display.

[0012] 1B illustrates an example of an augmented reality (AR) environment 100 in which an image dataset 116 of a patient 106 or person can be aligned with a real view of the patient 106 using an image visible marker or an image visible marker with an optical code attached to the patient 106. The environment 100 can include a physical space 102 (e.g., an operating room, a laboratory, etc.), a user 104, a patient 106, one or more image visible markers or an image visible marker with an optical code 200 of the patient, a medical tool 118, and an AR headset 108 in communication with a server 112 via a computer network 110. A virtual user interface 114 and a virtual cursor 122 are also shown in dashed lines to indicate that these virtual elements are generated by the AR headset 108 and are visible to the user 104 through the AR headset 108.

[0013] The AR headset 108 may be an AR computing system that can augment a real view of the patient 106 using the image dataset 116 or an image projection of the image dataset 116. For example, the AR headset 108 may be used by the user 104 to augment a real view of the patient 106 with one or more 3D image dataset views or radiological images of the patient 106, including but not limited to bones 106b (shown in FIG. 1), muscles, organs, cells, or fluids. The AR headset 108 may enable the image dataset 116 (or a projection of the image dataset) to be dynamically reconfigured. As the user 104 moves around the patient 106, sensors in the AR headset 108 determine the position of the user 104 relative to the patient 106, and the internal anatomical structure of the patient displayed using the image dataset 116 can be dynamically reconfigured as the user selects different orientations relative to the patient. For example, the user 104 may walk around the patient 106. The AR headset 108 may then augment the actual view of the patient 106 with one or more acquired radiological images or image datasets (MRI, CT scan, etc.) of the patient 106, so that both the patient 106 and the image dataset 116 of the patient 106 can be viewed by the user 104 from any angle (e.g., projection images or slices from the image dataset 116 can also be displayed). The AR headset 108 may be a modified version of the Microsoft HOLOLENS, Meta Company META 2, Epson MOVERIO, Garmin VARIA VISION, or other AR headsets.

[0014] The image visible markers or image visible markers with respective optical codes 200 may be attached to the patient 106 after generation of image data of the patient 106 (e.g. capture of MRI, CT scan, X-ray, etc.), but may remain attached to the patient 106 immediately before or while the patient 106 is viewed by the user 104 via the AR headset 108. The image visible markers or image visible markers with optical codes 200 may then be employed by the AR headset 108 to automatically align the image dataset 116 of the patient 106 with the actual view of the patient 106, as will be explained in more detail later. Furthermore, the image visible markers or image visible markers with optical codes 200 may be embedded with information for automatically acquiring the image dataset 116 and ensuring that the image dataset 116 acquired by the AR headset 108 matches the actual patient 106 displayed through the AR headset 108.

[0015] The virtual user interface 114 may be generated by the AR headset 108 and may include options to change the display of the projected internal anatomical structures of the patient 106 from the image dataset 116 of the patient 106. The virtual user interface 114 may include other information that may be useful to the user 104. For example, the virtual user interface 114 may include information about the patient or medical implement 118 (e.g., medical instruments, implants, etc.) that are identified with an optical code. In another example, the virtual user interface 114 may include medical charts or other medical data of the patient 106. In some configurations, the image dataset 116 or captured radiological data of the person may be displayed by the AR headset 108 using the volume of the image dataset 116 to display the radiologically captured anatomical structures of the patient 106 (e.g., bones 106b, tissues, blood vessels, fluids, etc.) from the image data. The image dataset 116 may include an axial slice, a coronal slice, a sagittal slice, or an oblique slice of the image dataset 116. The slices or image projections may be two-dimensional (2D) slices having depth as well as height and width (e.g., one or more layers of voxels), three-dimensional (3D) slices, and / or four-dimensional (4D) slices (3D images with a time series of images). The user 104 can control the virtual user interface 114 using hand gestures, voice commands, eye movements, a remote control (e.g., a finger clicker), a 3D mouse, a VR wand, finger sensors, haptic technology, or other control methods.

[0016] In one configuration, multiple users may be present at the same time, each wearing an AR headset 108, to view the patient 106 augmented with the image dataset 116 of the patient 106. For example, there may be multiple AR headsets 108 used during a medical procedure. One AR headset 108 may be used by a first medical professional to adjust and manipulate the radiological image displayed on both AR headsets, and a second headset may be used by a second medical professional to assist in performing the medical procedure on the patient. Additionally, one medical professional may be able to turn the radiological image on or off at the request of the other medical professional.

[0017] FIG. 2 shows an image visible marker 206 having an optical code 200 formed in a radiopaque material (e.g., metal or other radiopaque material) of the image visible marker, which can be attached to the patient 106. Also shown is an image visible marker 208 that is blank or does not include an optical code or other symbol. With reference to both FIG. 1 and FIG. 2, the optical code formed with the image visible marker can be perceptible to a sensor (e.g., an optical sensor, a camera, or an IR sensor) integrated into the AR headset 108. In some embodiments, the optical code 200 can be an AprilTag, a linear barcode, a matrix two-dimensional (2D) barcode, a quick response (QR) code, or some combination thereof. AprilTag is a type of two-dimensional barcode that can be a visual reference system useful for augmented reality and camera calibration. AprilTags can be used to calculate the 3D position, orientation, and identity of the optical code relative to a camera, sensor, or AR headset. Additionally, the optical code formed in the image visible marker or radiopaque marker can be in the shape of a cross, L-shape, image, letter, alphanumeric value, icon, etc., or any shape that can be formed in an image visible marker, or any shape that the image visible marker can be recognized by an AR headset. Similarly, an image visible marker without an optical code (e.g., a blank image visible marker) may be formed in the shape of a cross, L-shape, letter, alphanumeric value, icon, etc., or any shape that can be cast, printed, or otherwise formed and still be recognized by an AR headset.

[0018] The optical code 200 may further be associated with or combined with a marker 206 perceptible to a non-optical imaging modality. Examples of non-optical imaging modalities may include, but are not limited to, an MRI modality, a computed tomography (CT) scan modality, an X-ray modality, a positron emission tomography (PET) modality, an ultrasound modality, a fluorescence modality, an infrared thermography (IRT) modality, a 3D mammography, or a single photon emission computed tomography (SPECT) scan modality. In another example, the non-optical image or image dataset may be an image or image dataset that includes a combination of two or more forms of non-optical imaging modalities listed above (e.g., two or more images combined with each other, combined segments of two or more non-optical images, a CT image fused with an MRI image, etc.).

[0019] Forming the image visible marker 206 from a material perceptible to a non-optical imaging modality may allow the image visible marker 206 or an image visible marker with an optical code formed on the image visible marker 206 to appear in an X-ray generated image captured using an X-ray imaging modality. For example, the optical code may also be visible in an X-ray generated image. Examples of materials for the image visible marker 206 may include, but are not limited to, metal, radiopaque liquid, radiopaque plastic, metal impregnated rubber, metal strips, paramagnetic materials, and metal inks. If an optical code is used, the optical code may be formed directly in or on the image visible marker, such that the image visible marker and the optical code are joined together and can be acquired (e.g., visible) using an X-ray generated image and a sensor (e.g., a visible light camera or an IR camera) of an AR headset. For example, the optical code can be engraved, milled, or configured on a radiopaque material to make the optical code visible on the image visible marker in an X-ray imaging modality. Additionally, the metal may be configured to provide a visible optical code, or troughs may be formed in the metal to make the optical code visible in the metal (eg, titanium).

[0020] In yet another alternative, the image visible markers 206 may be embedded in the optical code 200 itself, for example, the image visible markers 206 may be embedded in the ink on which at least a portion of the optical code 200 is printed, the ink including a material that is visible to non-optical imaging modalities, such as radiopaque, x-ray opaque ink particles or additive printing manufacturing materials. In these embodiments, the optical code 200 itself may function as both an optical code and an image visible marker. Additionally, the image visible markers 206 may be placed by directly (at least temporarily) adhering or printing the optical code 200 comprising radiopaque ink particles onto the skin 106a of the patient 106.

[0021] Once the image visible marker 206 or the image visible marker with the optical code 200 is affixed to the patient 106, an x-ray imaging modality (in which the image visible marker 206 and the optical code 200 are perceptible) can be used to capture an x-ray generated image of the patient 106 and the image visible marker 206 or the image visible marker with the image visible optical code 202. As a result, the image visible marker 206 or the image visible marker with the image visible optical code 202 appears in an x-ray generated image of the patient 106, and the position of the image visible marker 206 or the image visible marker with the image visible optical code 202 appears in the x-ray generated image at a fixed position relative to the position of the internal anatomical structures of the patient 106.

[0022] A user 104 (such as a medical professional) can then use the AR headset 108 to determine the location of the image visible markers or image visible markers with optical codes 200 on the person's or patient's body. The AR headset 108 can then automatically retrieve an image dataset of the patient 106 based on the image visible markers or image visible markers with optical codes.

[0023] The X-ray generated image(s) and the image dataset may then be registered using data fitting. Further details regarding data fitting are given in the description of FIG. 3. The X-ray generated image(s) and the image dataset may be registered using anatomical structures (e.g., bones) seen in both images. In one example, the image visible markers 206 in the X-ray generated image or the image visible markers with the image visible optical code 202 may be copied to the image dataset, or both the X-ray generated image and the image dataset may be fused, so that both images have the image visible markers due to the image fusion. In the optional case where the image visible optical code 202 is used, the image visible code 202 may be copied to the image dataset along with the image visible markers.

[0024] Thus, the position of the image visible marker or image visible marker with optical code can be detected by the AR headset 108, and the position of the image visible marker(s) 206 or image visible marker with optical code is known in the 3D space 102 (see FIG. 1 ). After detecting the position of the image visible marker or image visible marker with optical code in the 3D space 102, the AR headset 108 may then align the position of the internal anatomical structure of the patient 106 in the 3D space 102 by aligning the image visible marker 206 in the 3D space 102 (as detected by the AR headset) with the position of the image visible marker 206 or image visible marker with image visible optical code 202 in the image dataset. This alignment may then enable the AR headset 108 to display the internal anatomical structure of the patient 106 in real time from the image data projected onto the actual view of the patient 106.

[0025] In a further example, multiple image visible markers 206 or image visible markers with optical codes 200 may be attached to the patient 106 simultaneously to further ensure accurate alignment of the image dataset of the patient 106 with the actual view of the patient 106 in the 3D space 102. Furthermore, because the image visible markers 206 or image visible markers with optical codes 200 are attached to the outer layer of the patient 106, the markers 206 are not all in one plane and can conform to any curvature of the outer layer of the patient 106.

[0026] FIG. 3 shows a more detailed schematic diagram of a method for registering an image dataset 310 with the body of a patient 320. An image dataset 310 may be acquired that includes a representation of the patient's anatomy and may not include image visible markers that may be used for registration purposes. This image dataset 310 may be a 3D image dataset and may represent a volume defined by the person acquiring the image dataset 310. The image dataset 310 may be a CT (Computed Tomography) image, an MRI (Magnetic Resonance Imaging), a 3D X-ray image, a Positron Emission Tomography (PET) image, a fluorescence modality, an Infrared Thermography (IRT) modality, a 3D mammography, or a Single Photon Emission Computed Tomography (SPECT) scan modality, a nuclear image, or an ultrasound image. In one example, the image dataset may be a CT scan captured as a maximum intensity projection (MIP) image, a mean intensity image, or a minimum intensity image.

[0027] A two-dimensional (2D) X-ray generated image 312 of at least a portion of the patient's anatomy in the image dataset 310 may also be acquired during or prior to the medical procedure. The image dataset 310 may be registered to the X-ray generated image 314 by using data fitting to register shared anatomy between the image dataset 310 and the X-ray generated image 314. In one configuration, an MIP version of a CT scan may be registered with the X-ray generated image. In particular, using an MIP version of the CT scan may provide similar pixel and / or voxel intensities in both the image dataset 310 and the X-ray generated image 312, which may improve the registration process.

[0028] Anatomical structures that may be used to register the image dataset 310 and the x-ray generated image 312 may include, but are not limited to: bone (e.g., spine, vertebrae, iliac crest, ribs, pelvis, or other bones), cartilage, skin, or organs identifiable in the x-ray generated image 310. These anatomical structures may have machine detectable structures in the x-ray generated image 310 that can be registered with the same machine detectable anatomical structures in the image dataset 312. Alternatively, a user or medical professional may mark the anatomical structures to be registered in the image dataset 312 or the x-ray generated image. The data fitting may be at least one data fitting process, such as using difference, squared difference, best fit, least squares best fit, least squares best fit using squared difference, curve fitting, regression analysis, or another known data fitting algorithm that allows for the registration of shared anatomical structures in the image dataset 312 with the x-ray generated image 310 or vice versa. In one configuration, at least two x-ray generated images may be acquired to allow depth registration using one or more radiopaque markers and optionally their optical codes.

[0029] As previously mentioned, an optical code can be formed in the metal of the radiopaque marker. The pattern of the optical code in the metal of the radiopaque marker can be formed using troughs in the metal, or the shape of the optical code can be raised metal. In either case, the different thicknesses or 3D protrusions of the optical code can be seen or detected in the X-ray generated image and can be detected in the visual or IR image acquired by the AR headset. In an alternative configuration, the radiopaque marker can be made of metal with an optical code in the metal, but also have a printed optical code (e.g., 2D on paper or plastic) deposited on the metal. The radiopaque marker can also have various shapes (e.g., square, triangle, circle, rectangle, etc.) for identification purposes. The optical code formed in the radiopaque marker can be a linear barcode, a matrix two-dimensional (2D) barcode, a quick response (QR) code, or an AprilCode. Furthermore, the optical code formed on the image visible marker or radiopaque marker may be in the shape of a cross, L-shape, image, letter, alphanumeric character, icon, etc., or any shape that can be formed on the image visible marker or that can be recognized by the AR headset. It is useful that the optical code is detectable in the visual or IR range to enable detection by the sensor of the AR headset, but also detectable in non-visible imaging modalities (e.g., X-ray) that can be used for alignment during medical procedures. Forming the optical code in the image visible marker or radiopaque marker or from a radiopaque material means that the optical code spatially coincides with the image visible marker.

[0030] The positions of the image visible markers or the image visible markers with optional optical codes can be defined in the image dataset 310 after registration with the X-ray generated image 312. For example, the image visible markers or the image visible markers with optical codes 322 in the X-ray generated image 312 may be copied to the image dataset 312 after the image dataset 310 and the X-ray generated image 312 are registered. Alternatively, the image dataset 312 and the X-ray generated image 312 may be fused, which means that the image visible markers or the image visible markers with optical codes may be present in the fused image.

[0031] The image dataset 314 and / or the X-ray generated image 312 may then be aligned with the body of the patient 320 based in part on image visible markers or image visible markers having optical codes 322 defined within the image dataset 314, with reference to image visible markers or image visible markers having optical codes 322 that are visible on the patient through the AR headset.

[0032] A point location for an image visible marker or an image visible marker having an optical code on the patient's body in the 3D coordinate system may be identified by calculating a ray from the x-ray source through the center of the radiopaque image visible marker in each of the at least two x-ray generated images 310. The image visible marker is then determined to be at the intersection of the at least two calculated rays. Further, the location of the image visible marker on the patient's body can be identified by calculating a ray from the x-ray source through the center of the radiopaque marker in the x-ray generated images and determining that the radiopaque marker is at the intersection of the ray and the visible skin.

[0033] Another method for determining the location of the image visible marker or image visible marker with an optical code may use a single x-ray, where multiple points on a single radiopaque marker are identified and then the field of view (FOV), focal length, or focus of the x-ray may be known. These points on the radiopaque marker and x-ray attributes can be used to calculate a PMP (fluoroscopic multipoint) calculation and determine the location of the image visible marker or image visible marker with an optical code.

[0034] The registered image data set may then be displayed as a superimposition 318 on the body of the patient 320 as visible through the lenses (e.g., waveguide lenses) of the AR headset. In one configuration, as described, the X-ray generated image 310 and the image data set 312 may be fused and the X-ray generated image 312 and the image data set 312 may be displayed using the AR headset and image visible markers or image visible markers with optical codes 322 to form a single graphical view as a superimposition on the patient's body as visible through the lenses of the AR headset.

[0035] The present technology allows for the registration of an image data set with a person's body even when the image data set is captured without an image visible marker or an image visible marker with an optical code for registering the image data set with the person's body. In one example, a medical professional can obtain a CT scan previously captured without an image visible marker or an image visible marker with a corresponding optical code. The image visible marker or the image visible marker with an optical code can be added to the patient before the medical procedure. The image visible code or the image visible marker with an optical code can then be identified and registered by the AR headset.

[0036] Since the patient was scanned without an image visible marker or without an image visible marker with an optical code, in the operating room, the medical professional can take one or more 2D X-ray projections of the patient in the operating room. A radiopaque shape (e.g., a square) or an image visible marker with an optical code formed thereon can be placed on the patient in the operating room and captured in the 2D X-ray projection. For example, the metal shape may be a titanium square that is visible in the X-ray produced image. If an optical code is formed on the radiopaque shape, the X-ray projection can capture the radiopaque shape along with the optical code pattern engraved or formed on the metal shape. Registering and / or merging the X-ray produced image 312 with the CT image scan 310 allows for the calculation of 3D coordinates using the X-ray produced image that define where the image visible code 322 is in the CT image space. Shared anatomical structures in the X-ray produced image and the image dataset (e.g., the patient's spine, iliac crest, sternum, etc.) allow for the registration of these two images.

[0037] In a more detailed registration example, the CT scan can capture internal anatomical structures such as the patient's spine, iliac crest, or sacrum. The CT scan can then be rendered as a MIP (maximum intensity projection) image. This makes the MIP similar in intensity to the X-ray generated image, and bones can be better identified for the registration process using the MIP. Image visible markers or image visible markers with optical codes can be placed on the patient in the operating room, and the X-ray generated image may be captured while the patient is in the operating room. In the case of the vertebrae example, both the vertebrae and the radiopaque code are visible in the X-ray generated image. The goal is to align the CT scan (e.g., a hologram) to the actual patient as seen through the AR headset, even if the image visible markers are not present in the patient's previous CT scan.

[0038] Bones or similar elements in the CT scan that are easily identifiable in the X-ray generated image can allow the CT scan and the X-ray generated image to be mapped together using common anatomical structures. The image visible markers or image visible markers with optical codes that are visible in the X-ray generated image can be transferred to the image dataset to perform alignment of the image dataset with the patient. The image visible markers may be blank titanium tags or titanium tags with optical codes (only engraved or printed) that are visible to the human eye and to the AR headset. The blank titanium tags or titanium tags with optical codes may be visible in the X-ray generated image (engraved or constructed metal pattern). This allows the technology to indirectly learn (e.g., copy) where the image visible markers or image visible markers with optical codes should be on the original CT scan (e.g., the 3D coordinate system or reference frame of the CT scan).

[0039] Referring again to FIG. 3, the X-ray generated image 312 cannot be rotated in three dimensions or viewed from different virtual camera viewpoints. The CT scan 310, on the other hand, is a 3D image dataset that the user can rotate and view from different rendered viewpoints on an AR headset or a 2D screen. The fusion of the X-ray generated image 312 and the CT scan 310 is shown as a combined image 314, where the light areas in the image are the difference between the corresponding images at each location, indicating the "squared difference" between the images. More specifically, the combined image shows that most of the spine is offset and dark because the spine is in both images and aligned. This alignment is performed as a "best fit" alignment using an AR headset during a medical procedure. The alignment process aligns bones or anatomical structures such as the spine, iliac crest, disc, spinal cord, organs, and / or other structures. For example, the spine can be aligned and / or the iliac crest may be partially aligned. The registration process can be performed by moving (e.g., translating or rotating) two or more images (CT scan images and X-ray generated images) relative to one another until they are as accurately registered as possible. If the shared anatomical structures are highly aligned between the CT scan images and the X-ray generated images, the registration process determines that the CT scan is in the same position as the X-ray generated images were taken, at the same angle as the X-ray generated images were taken, at the same magnification as the X-ray generated images were taken, etc. The system can use orthogonal or perspective projections of the CT scan (e.g., closer anatomical structures are larger and more distant anatomical structures are smaller) for this registration process.

[0040] By capturing radiopaque image visible markers or image visible markers with optical codes in the x-ray generated images, the image visible markers or image visible markers with optical codes can be linked to the patient's internal anatomical structures (e.g., lumbar spine, pelvis, ribs, etc.). Fusing the two images using the bony anatomical structures is the shared registration linking factor. Thus, the image visible markers or image visible markers with optical codes are tied to the patient's anatomical structures (e.g., bones), and the registration is further tied to anatomical structures that are visible in both the x-ray generated images and the image data set.

[0041] 4A shows an example of a human body 402, a human iliac crest 404, and an illustrated spine 408. The registration process can determine where the x-ray generated image matches on the human body. The image visible marker or image visible marker with optical code 420 may be in the x-ray generated image at a defined location, and the center of the optical code can be found using the intersection of the diagonal or center of gravity of the optical code. The image dataset 406 registered with the x-ray generated image using bones (e.g., spine 408, iliac crest 410, or sacrum 412) can then be registered with the human body 402 using the image visible marker or image visible marker with optical code 414 in the fused image (of the x-ray and image dataset), or image visible marker copied to the image dataset 406.

[0042] FIG. 4B shows that the position of the image visible marker or image visible marker with optical code can be determined on the patient using a ray 450 (e.g., a mapping ray) defined from a known position of an X-ray source 452 to an X-ray detector 462. The ray 450 passes through the center of the image visible marker 458 or image visible marker with optical code to the patient and can pass through the body if desired (not shown). When the mathematical construction of the ray is created, it is known that the image visible marker is on the ray or line, but the system does not know exactly where the image visible marker or image visible marker with optical code 458 may be on the ray relative to the body. However, the position where the ray intersects with the skin may be known when the image visible marker or image visible marker with optical code is placed on the skin.

[0043] If the image visible marker or image visible marker with optical code is not on the skin, one or more additional 2D X-ray generated images can be captured from at least two perspectives. For example, the first X-ray generated image 454 can be from a first view, the second X-ray generated image 456 can be 45 degrees from the first view, etc. As described above, the X-ray source position is used to find the location of the titanium image visible marker or image visible marker with optical code. The X-ray generated image can be a projection of the person's body from the X-ray source. The rays or projection from at least a second angle allows the location of the image visible marker or image visible marker with optical code 460 to be determined without assuming that the code is on the skin. After two or more X-ray generated images from separate perspectives are acquired, the location of the image visible marker or image visible marker with the corresponding optical code can be found using triangulation. In one example, the image visible marker may be placed on an internal anatomical structure (bones, organs) and may not be on the skin.

[0044] Once the locations of the image visible markers have been identified in 3D coordinates (e.g., x, y, and z Cartesian coordinates) using the two X-ray generated images, the system can identify where each image visible marker is located within the image dataset to enable alignment of the image dataset with the person's body. Again, rays can intersect with the skin to identify the location of the image visible marker or image visible marker with an optical code, or rays representing one or more additional X-ray projections can be triangulated to define where the image visible marker is within the 3D coordinate space viewed by the AR headset.

[0045] In practice, the patient's CT scan may have been previously captured without image visible markers or with image visible markers with optical codes. Instead of spending time to recapture a CT scan with image visible markers or with image visible markers with optical codes before surgery, the patient can arrive at the surgery facility and have two x-rays taken. The image data set can then be aligned with the person's body, followed by the medical procedure. The use of two-dimensional x-ray exposure in this technique is beneficial because x-ray imaging equipment (e.g., x-ray C-arms, etc.) is less expensive than CT or MRI scans, is widely used, and is more convenient and available than other imaging modalities (e.g., CT scans or MRIs).

[0046] When acquiring additional X-ray generated images, the C-arm can be used to rotate around the person's body to acquire the additional X-ray generated images. In some cases, the additional X-ray generated images can be generated using a different angle than the original X-ray. When the 2D X-ray generated image and the 3D image data set are registered using data fitting, the two images are registered in X and Y coordinates, and the image can be rotated (along the spine or other marker) to further align the CT scan with the 2D X-ray generated image. This process can reconstruct the CT scan from any view or projection, and then the CT scan can be registered with the 2D X-ray generated image.

[0047] 5 illustrates a technique 502 that uses an augmented reality (AR) system to display an X-ray generated image 554 and / or an image projection 556 from an image dataset in registration with a body 506a of a person 506. A fluoroscope 560 can transmit an X-ray beam (e.g., continuous X-rays or single exposures) through the patient to obtain one or more X-ray generated images or live video of the fluoroscopic imaging of the patient that is visible to a medical professional. The fluoroscope 560 may also be movable relative to the person's body.

[0048] The image projections 556 may include or represent at least one anatomical structure within the patient's body. The image data set may also be a CT scan captured as a maximum intensity projection (MIP) image, a mean intensity image, or a minimum intensity image.

[0049] The x-ray generated image 554 may include at least a portion of the anatomical structure in the image dataset. The anatomical structure shared between the image dataset and the x-ray generated image 554 may be bone, cartilage, skin, or organs that are visible in the x-ray generated image and the image dataset. The x-ray generated image 554 may also include image visible markers, or image visible markers with image visible optical codes generated from optical codes made of metal or radiopaque material, or optical codes formed in metal or radiopaque material. In one configuration, two or more x-ray generated images may be used to enable alignment of the image dataset with the human body using x-ray generated images and image visible markers or image visible markers with optical codes.

[0050] The image dataset may be registered to the x-ray generated image 554 by using data fitting to register shared anatomical structures (e.g., bones) in the image dataset (e.g., image projections 556) and the x-ray generated image 564. Data fitting may be performed using image difference, squared difference, best fit, least squares best fit, least squares best fit using squared difference, curve fitting, regression analysis, or another known data fitting technique.

[0051] Once the image projection 556 from the image dataset and the x-ray generated image 554 are aligned, the position of the image visible marker or the image visible marker with the optical code can be defined in the image projection 556 from the image dataset based on the alignment with the x-ray generated image 554. As discussed above, the image visible marker or the image visible marker with the optical code can be combined with each other in a titanium marker with an optical code formed in the image visible marker. In one configuration, the system can copy the image visible marker or the image visible marker with the optical code from the x-ray generated image to the image dataset. Alternatively, both the image dataset and the x-ray generated image can be fused into a single image.

[0052] The image projections 556 from the image dataset may then be aligned with the patient's body using image visible markers, or image visible markers having optical codes visible on the patient anatomy, and said image visible markers defined in the image projections 556 from the image dataset (or fused image).

[0053] The orientation and position of the X-ray imaging C-arm can be determined using a number of optical codes 552, 558 on the X-ray imaging C-arm. Thus, the orientation and position of the X-ray generated image 554 captured by the X-ray imaging C-arm is also known. The orientation of the C-arm in 3D coordinate space relative to the AR headset or the patient from which the X-ray generated image 554 is captured can be determined. Slice, slab, or 3D volume projections of image projections 556 from the image dataset may also be configured in a plane that aligns with the orientation and position of the plane of the X-ray generated image 554 in the 3D coordinate space at a defined thickness, visible through the AR headset. The image projections (e.g., slices) and / or X-ray generated images may be displayed to form a graphical view as a superimposition on the patient anatomy using the AR headset. Alternatively, the image projections may be displayed on a video monitor 562.

[0054] An alternative method for detecting the position of a mobile fluoroscope (e.g., a C-arm) is to use sensors in the AR headset to detect the shape of the mobile fluoroscope or C-arm. These sensors on the AR headset may be a visible light camera, an IR sensor, a LIDAR, or any sensor that can be used with software to detect the shape, position, and orientation of the C-arm in a 3D coordinate system seen by the AR headset. Such automatic detection of the C-arm by shape can help avoid or reduce the use of an optical cord on the C-arm.

[0055] In a more specific example, the x-ray generated images 554 and image projections 556 can have positions and / or orientations defined by the fluoroscope 560 and / or image visible markers. A camera or optical sensor (e.g., visible light sensor or IR sensor) linked to the AR system or AR headset 508 can capture visual image data of a portion of the person's body on the operating table 503, and the fluoroscope 560 that is movable relative to the person's body. One or more optical codes on the person's 506a body and one or more optical codes on the fluoroscopes 552, 558 can be identified and scanned from the visual image data using optical code recognition and scanning technology or other optical code recognition technology.

[0056] The image projection 556 may be selected to display the portion of the image data set associated with the X-ray generated image 554 (e.g., parallel, oblique, or in another fixed orientation) that is captured. The image projection 556 may also display certain anatomical structures of the person's body, such as veins, nerves, or bones. The X-ray generated image 554 may be a single projection (e.g., a two-dimensional (2D) projection). The registration, positioning, and orientation may be performed using image visible markers or image visible markers with optical codes 520 on the person's body, a representation of the image visible markers copied to the image projection as described above, image visible markers or image visible markers with optical codes captured in the X-ray generated image 554, and optical codes 552, 558 on the fluoroscopy device (e.g., a C-arm device, a catheterization lab, an angiography lab, or a fluoroscopy device with a movable emitter and detector).

[0057] Image projections 556 are projected (e.g., multi-layer projections) from the image dataset to define a single planar view or "slab" that is a multiplanar reconstruction (MPR) of the image dataset. The image projections may be of any selected thickness and may be MIP (maximum intensity projection) of the tissue in the appropriate view. Image projections that are not aligned with the fluoroscope orientation may also be made at any position, orientation, or angle, as desired.

[0058] At least one of the image visible markers or image visible markers with optical codes 520 may be on the person's body (as previously described in detail). This allows an image dataset (e.g. a captured radiological image) with image visible markers or multiple image visible markers with optical codes copied from an X-ray generated image to be registered with the body of the person 606 using the image visible markers or image visible markers with optical codes on the person's body. Parts of the person's body may be covered with a cloth 507, but the internal anatomical structure 550 of the body can be virtually viewed using the image dataset.

[0059] As previously mentioned, multiple useful views can be provided to the medical professional using the fluoroscope and AR system. In one view, the projections from the image dataset can be moved or reconfigured to match the medical professional's changing actual view of the body through the AR system or AR headset. Yet another view can be provided with an AR system that displays a combined or fused view of an X-ray generated image and a projection of the image dataset (e.g., a 2D rectangular slice) parallel to the X-ray generated image superimposed in registration with the patient (e.g., what the medical professional would see if he or she were in the same perspective as the X-ray beam itself). In this configuration, the projections can be moved or reconfigured as the fluoroscope moves. The X-ray generated image 654 can be set to any level of transparency desired by the medical professional or user. In another configuration, the reconstructed slices of the image dataset can be constructed in real time or immediately after 2D X-ray generated images are captured of the patient or person's body.

[0060] The registration process of the present technology can help reduce the amount of radiation used in treating a patient. Because a high contrast resolution CT scan can be registered to the patient, more detailed aspects of the patient's anatomy can be seen with less dye or radiation used in the treatment procedure. While x-ray generated images do not allow medical professionals to see finer details of the patient's anatomy (except for bones), x-ray generated images allow for a live view that is useful for guiding radiopaque medical tools. In comparison, image datasets allow for finer details of the patient's anatomy to be seen, but need to be registered to the person.

[0061] The image dataset may also be segmented such that certain parts of anatomical structures and / or organs can be turned off or on and viewed from the image dataset. For example, an x-ray generated image may not show the patient's blood vessels, but the segmented image dataset may only show blood vessels that have been identified as segments in the image dataset. In another example, the image dataset may be segmented by organ or anatomical type, so that the user or medical professional may audibly request the computer and AR headset to view certain anatomical structures. The user may request the software process on the AR headset to hide or view bones, muscles, or other anatomical items in any combination or at any time.

[0062] FIG. 6 illustrates that the position of the fluoroscope 560 may change relative to the person 506 or patient's body 506a on the table 503 to allow the medical professional 504 to acquire an X-ray generated image 554 from a different perspective. The change in the position and orientation of the fluoroscope relative to the person's 506 body 506a may be detected and quantified using one or more optical codes 552, 558 captured by the AR headset 508 or a camera associated with the AR system. The change in the position and orientation of the fluoroscope 560 may alter the position and orientation of the image projection 556 and the X-ray generated image 554. For example, the position and / or orientation of the image projection 556 from the image dataset may be moved based on the detected change in the position and orientation of the fluoroscope 560 using one or more optical codes 552, 558, as seen by the AR headset, relative to the patient's 506a body. The image projection 556 from the image dataset may be reconstructed using the modified position and orientation of the fluoroscope 560, or a new projection may be created.

[0063] For example, the fluoroscope 560 may be rotated 45 degrees in one axis. As a result, the image projection 556 is recreated in its new orientation, the X-ray generated image 554 is displayed in a rotated orientation as viewed on the AR display, and the orientation of the X-ray generated image 554 and the image projection 556 can be aligned in the proper orientation relative to the person's body 506a as viewed through the AR headset. The X-ray generated image 554 can have a modified orientation in 3D space relative to the person's body as defined by an optical code, image visible markers on the body, and / or as defined by a modified position and / or orientation of the fluoroscope 560. Thus, as the position and orientation of the X-ray beam changes, the position and orientation of the image projection 556 and the X-ray generated image 554 can change. Thus, the optical tags on the C-arm can control how the image projections or slabs are reconfigured as the C-arm moves. For example, the CT scan can be reconstructed to match the X-ray generated images that the C-arm is generating, and the CT and X-ray generated images can be registered and / or fused to allow the medical professional to see the fine anatomical structures (e.g., blood vessels) that are being targeted.

[0064] Determining the position and orientation of the fluoroscope 560 relative to the patient also allows the AR system to reconstruct the image projections 556 (if desired) so that they are parallel to the x-ray generated images 554 obtained from the fluoroscopy detector, if necessary. Additionally, the x-ray generated images 554 can be positioned relative to the body of the person 506a based on the location where the x-ray generated images 554 are captured from the body using the x-ray beam. Thus, the x-ray generated images 554, the image projections 556, and the patient's body 506a can be aligned, and a medical professional can use the image projections 556 as a superimposition on the person's body (e.g., above or below the x-ray generated images in a 3D coordinate system) to view the person's or patient's anatomy. The positioning and orientation of the image projections 556 and the x-ray generated images 554 can represent an AR (augmented reality) view based on the portion of the person's body through which the x-ray beam is passing (as opposed to the medical professional's point of view). In this case, the image projections 556 may be perpendicular to the x-ray beam.

[0065] The AR system can reconstruct the 3D image dataset to provide projections from any angle consistent with the position and orientation of the fluoroscope 554. For example, if the fluoroscope 560 is positioned to capture a lateral view, a lateral projection of the image dataset can be provided along with a lateral X-ray generated image view. By combining and registering the real-world view of the person's body, the image projections from the image dataset, and the X-ray generated image, medical professionals can better see and navigate the patient's internal anatomy. The 3D (three-dimensional) image dataset provides better soft tissue contrast (e.g., organs and blood vessels can be seen in the internal anatomy) and a 3D reference environment that X-ray generated images cannot provide.

[0066] In one configuration, the fluoroscope 560 can be zoomed relative to the body of the person 506 while capturing the x-ray generated image 554. As a result, adjustments to the image projection 556 or image dataset can be performed based on the zoom in the x-ray generated image. More specifically, once the x-ray generated image is aligned with the image projection or image dataset (even if the relative sizes when aligned are not exactly the same), this alignment can be used to correct for zoom issues by increasing or decreasing the size of anatomical structures in the image projection or image dataset until the size of the anatomical structures matches the x-ray generated image. The anatomical structures in the image become reference markers for zooming the image dataset.

[0067] The use of the AR display may also provide the ability to change the view of the X-ray generated images and image projections to be displayed aligned on the AR display in order to display the virtual patient to better match the position and orientation of the actual view of the patient as viewed directly by the medical professional. Regardless of the orientation of the X-ray generated images captured by the fluoroscope, the display of the X-ray generated images and image projections may be oriented in the AR display to assist the physician and / or match the actual view of the patient's position and / or orientation. For example, the X-ray generated images may be flipped horizontally, vertically, or anisotropically in some other manner compared to the patient's body as viewed in reality. The difficult viewing direction may be due to the capture direction of the fluoroscope 560. Thus, the image with the aligned X-ray generated images 554 and image projections 556 may be flipped or reoriented (e.g., flipped horizontally, flipped by a certain angle, moved to reverse the misaligned orientation) for viewing by the medical professional to make the medical procedure easier to perform or more closely match what the medical professional sees in the actual view. This ability to change the image orientation allows for more intuitive interaction when performing a procedure on the patient. For example, performing a medical procedure with X-ray generated image guidance can be very difficult when everything is posterior.

[0068] 7 shows a side view of a cross-section of a body of a person 706, a combination of image projections 704 from a registered image dataset 720, a registered X-ray generated image 702, and a fluoroscope 714, which may enable a medical professional to fluoroscopically navigate a visible item 710 within the body of the person 706. A see-through object 710 (e.g., a needle) may be seen and guided by a medical professional within the X-ray generated image 702 relative to the image dataset 720 and / or the image projection 704 registered with the body of the person. The image projection 704 may appear superimposed on the X-ray generated image 702 in an AR headset or AR display. Alternatively, the image projection 704 may appear as if the X-ray generated image 702 is superimposed on the image projection 704, or the image projection 804 may appear to be within the registered image dataset 720. The registration of the image dataset 720 and its image projections 704 with the patient may be performed using the X-ray generated image, as described above.

[0069] The transparency of the X-ray generated image 702 registered with the image dataset can be changed depending on how much of the image projection 704 or real-world body of a person 706 the medical professional wishes to see.

[0070] The image projections 704 from the image dataset 720 may have been captured using computed tomography (CT) images or magnetic resonance imaging (MRI). The image projections 704 may then be overlaid with a live X-ray generated image 702. Although the X-ray generated image 702 may be a live image, the X-ray generated image does not have the 3D quality or soft tissue contrast resolution of the MRI (magnetic resonance imaging) or CT (computed tomography) images. When the X-ray generated image 702 is registered to the patient's body using one or more image visible markers (such as AprilTag) or image visible markers with optical codes, and the image projections 704 of the previously acquired 3D image dataset are registered to the image visible markers or image visible markers with optical codes on the patient's body, the medical professional can use an AR headset or AR display to see the virtual end 710 of the needle as the tip of the needle moves within the patient's 706 body. This combines valuable aspects of virtual images, X-ray generated images, and a real view of the patient. Although needles, catheter tips, or similar radiopaque objects can be seen under fluoroscopy, the medical professional cannot see certain soft tissues in the x-ray generated image. Thus, the combination of an actual view of the patient, an image data set, an x-ray generated image, an optical tag on the fluoroscope, an image visible marker, an image visible marker with an optical code, the medical tool, and an optical tag on the medical tool may enable the medical professional to perform a more precise medical procedure.

[0071] In a more specific example, the image dataset may be a CT image that was originally captured without image visibility code. The CT image may provide a 3D view of the voxel slice data captured in the CT scan. MIP rendering may also be used to generate the CT image. A portion of the CT image may be selected that is a projection of a slab of tissue, providing an image that looks similar to an x-ray of the spine (i.e., bone). The protrusion may have a thickness of a portion of the patient's anatomy. For example, the slab may have a thickness of a spine.

[0072] If the CT scan has skin but includes bones such as the spine, the skin can be used to identify the location of the optical code. Additionally, two or more x-ray generated images or consecutive x-rays can be used to identify the optical code at a location other than the skin by using triangulation, and then registration can be performed without the skin in the CT scan. In another configuration, one x-ray generated image may be registered to the person's body in the X and Y planes, and then an image dataset or view of the skin through the AR headset may be used to determine the location of the visible markers or optical code in the Z direction using the patient's skin.

[0073] When a medical professional is performing an angiogram, the patient can be viewed with continuous x-ray or fluoroscopy. When the medical professional rotates the beam of the C-arm during an angiogram, the optical code on the C-arm can allow the x-ray generated images and the MIP projections from the CT scan to have the same orientation and similar positioning.

[0074] FIG 8A shows a projection of the CT scan 704 onto the vertebral bodies rotated 90 degrees compared to FIG 7 as the C-arm has moved 90 degrees. FIG 8B shows a projection of the CT scan 704 moved 45 degrees compared to FIG 7. As previously explained, the spine or vertebrae can be used to align the CT scan 704 with the anatomical structures (bone structures) in the CT scan 704 with the anatomical structures (e.g., the same bony structures) in the x-ray generated image 702, and the CT scan can then finally be aligned with the patient. This alignment can be done in real time during a medical procedure.

[0075] Projections 704 from the slab of tissue in the CT scan 720 can be mapped to the same location that the X-ray generated image 702 captures. The CT scan can show detailed internal tissue such as blood vessels or other tissues (aorta, blood vessels). As a result, the medical professional can find this tissue or blood vessel on the CT scan before the start of the medical procedure or during the medical procedure. Because the fluoroscope captures a single plane projected through the patient and the 2D X-ray generated image has very limited contrast and does not show soft tissue well, the use of image projections 702 of the CT scan 720 registered with the X-ray generated image 702 improves accuracy during the medical procedure. Once the CT scan 720 is joined to the correct location defined by the 2D X-ray generated image, the medical professional can obtain a 3D view of any anatomical structures (e.g., bones, blood vessels, organs, nerve tissue, etc.) visible in the CT scan.

[0076] In some medical procedures, medical professionals may use injections of dye to help locate locations within blood vessels. However, using too much dye to insert a catheter or other medical tool into a patient can damage or distress the patient's kidneys. Using this technique, blood vessel locations can be located using a CT scan superimposed on the patient, increasing the contrast resolution of the CT scan 720 compared to the 2D X-ray generated image 702. This technique can provide real-time guidance or visualization (e.g., a roadmap) for a medical procedure. The X-ray generated image may allow the medical professional or the user of the AR headset to see the catheter or other radiopaque tool in real time, while the CT scan (i.e., image dataset) may allow the medical professional to see more detailed anatomical features when correctly aligned. This process reduces the amount of dye in the patient and the radiation used while the medical professional performs the medical procedure.

[0077] 8B and 8C further illustrate that the image projections from a CT scan or entire CT scan can remain registered using the present technique regardless of the orientation of the mobile x-ray device (e.g., C-arm). For example, if a medical professional rotates the C-arm, bones or blood vessels can be seen from a different orientation and the image projections from the CT scan 720 can be matched to the orientation of the C-arm. Optical codes on the C-arm and on the patient allow the x-ray generated image 702 and the CT scan data 720 to be registered together. Automatic reconstructions of the CT scan 720 can be generated in frontal, oblique, side, or any projection as the mobile imaging device moves.

[0078] In another configuration, the system can start with an X-ray generated image and then automatically create the desired image projections from a CT scan (e.g., an image dataset). The image projections of the image dataset can represent a slab of tissue including identifiable elements within the slab, such as the spine, iliac crest, pelvis, etc. The X-ray generated image can also have corresponding identifiable anatomical elements (e.g., spine, iliac crest, bones, etc.). Thus, an image projection from the image dataset may be selected based on a known anatomical element to be highlighted or acted upon. Alternatively, a medical professional may select or highlight a particular anatomical structure (e.g., spine, vertebrae, pelvis, etc.), which may determine the projections to be generated.

[0079] The X-ray generated images can define the orientation and position for generating useful projections from the image data set. The technology can automatically align the image visible and optical codes on the patient, the mobile fluoroscope, and the orientation and position of the X-ray generated images, and then the system can find the best-fit projection from the CT images. In these examples of alignment, the bones become the internal alignment system of the images, and the image visible optical codes become the external alignment system.

[0080] The thickness of the slab for the image projection taken from the image dataset (e.g., CT scan) may be selected by a medical professional. In addition, the medical professional may determine the depth of the slab in the image dataset in any direction. The location of the X-ray generated image displayed relative to the person's body in the AR headset may also be determined. For example, the X-ray generated image may be displayed to appear outside the person's body. Alternatively, the X-ray generated image may be displayed near or adjacent to the image projection from the image dataset, as shown in Figures 8B and 8C. In addition, automatic detection of the slab location or depth may be around a particular bone, organ, depth of an anatomical structure, estimated depth of an anatomical structure, etc.

[0081] 9A is a flowchart of an exemplary method for aligning an image dataset with a patient's body, where the image dataset can be displayed using an augmented reality (AR) headset worn by a medical professional. In these and other embodiments, method 900 may be performed by one or more processors based on one or more computer-readable instructions stored on one or more non-transitory computer-readable media.

[0082] The method may include acquiring an image dataset representative of a patient's anatomy (e.g., bone, cartilage, skin, etc.), as in block 902. For example, the image dataset may be a CT scan or an MRI scan captured as a maximum intensity projection (MIP) image. Similarly, a two-dimensional (2D) X-ray generated image of at least a portion of the patient's anatomy in the image dataset may be acquired, as in block 904.

[0083] Another operation may be to register the image dataset to the x-ray generated image by using data fitting to register anatomical structures (e.g., bone, cartilage, skin, or organs) shared in the image dataset and the x-ray generated image, as in block 906. The data fitting used between the two images may be difference, squared difference, best fit, least squares best fit, least squares best fit using squared difference, curve fit, regression analysis, or another data fitting technique to match a bone or similar identifiable tissue clearly visible in the x-ray generated image to the same bone or tissue in the image dataset. In one configuration, at least two x-ray generated images may be acquired using radiopaque markers (i.e., image visible markers) or image visible markers with optical codes to enable registration in the depth direction.

[0084] The location of the radiopaque markers or radiopaque markers with optical codes in the image dataset may be defined using registration with the x-ray generated images, as in block 908. For example, one or more image visible markers with optical codes representing the radiopaque markers in the x-ray generated images registered with the image dataset may be copied to the image dataset.

[0085] The image data set may then be registered with the patient's body based in part on the radiopaque marker or radiopaque marker with an optical code defined in the image data set relative to the radiopaque marker or radiopaque marker with an optical code, as in block 910. The location of the center point (or other coordinate point in 3D coordinate space) of the radiopaque marker on the patient's body may be identified by calculating a ray from the x-ray source through the center of the radiopaque marker in each of the at least two x-ray generated images and determining that the radiopaque marker is at the intersection of both rays (i.e., triangulation). In addition, the point of the radiopaque marker on the patient's body may be identified by calculating a ray from the x-ray source through the center of the radiopaque marker in the x-ray generated image and determining that the radiopaque marker is at the intersection of the ray and the patient's skin. The triangulation of the rays may be used to identify image visible markers or image visible markers with optical codes that are not on the skin of the patient's body.

[0086] The optical code may be formed within the material of the radiopaque marker (e.g., metal or radiopaque material). In another configuration, the radiopaque marker may be metallic and have the optical code applied to the metal (e.g., metallic ink on plastic or paper). The optical code may be a linear barcode, a matrix two-dimensional (2D) barcode, a quick response (QR) code, or an AprilCode.

[0087] A further operation may be to display the image dataset or image projections from the image dataset as a superposition on the patient's body, which is visible through the lens of the AR headset. The alignment and compositing of these multiple image aspects is visible through the AR headset. The X-ray generated images can optionally be displayed together with the image dataset using the AR headset to form a single graphical view as a superposition on the patient's body, which is visible through the lens of the AR headset.

[0088] 9B is a flow chart illustrating another method for aligning an image dataset representing medical imaging of a patient with a patient's body using an augmented reality (AR) headset worn by a medical professional. In particular, the method involves the use of a mobile x-ray imaging device and the generation of projections or slices of the image dataset using the orientation of the mobile x-ray imaging device.

[0089] The method may include acquiring an image dataset having the patient's anatomy, as in block 952. For example, the image dataset may be a CT scan or similar medical image captured as a maximum intensity projection (MIP) image. An X-ray generated image may also be acquired, as in block 954, that includes at least a portion of the anatomy in the image dataset.

[0090] Another operation is to register the image dataset to the X-ray generated image by using data fitting to register anatomical structures shared between the image dataset and the X-ray generated image, as in block 956. The data fitting may be a least squares best fit using the squared difference. This registration of the two images allows the location of the radiopaque markers or radiopaque markers with optical codes to be defined in the image dataset using the registration with the X-ray generated image, as in block 958. For example, the location of the radiopaque markers or radiopaque markers with optical codes may be copied from the X-ray generated image to the image dataset. In one configuration, at least two X-ray generated images are used to enable the registration of the X-ray generated image and the image dataset using the radiopaque markers.

[0091] A further operation may be to register the image data set with the patient's body. Registration may be performed using radiopaque markers or radiopaque markers with optical codes visible on the patient's anatomy, and radiopaque markers or radiopaque markers with image visible optical codes may also be defined in the image data set, as in block 960. The orientation and position of the X-ray imaging C-arm may be determined using multiple optical codes on the X-ray imaging C-arm, as in block 962. Since the X-ray generated images are captured by the X-ray imaging C-arm, the orientation and position of the X-ray generated images may also be known.

[0092] The orientation and position of the x-ray imaging C-arm may be used to construct a projection or slice of the image dataset in the plane based in part on the orientation and position of the plane of the x-ray generated image, as in block 964. The projection or slice may have a thickness defined at selected voxels or layers from the image dataset. Finally, the slice and x-ray generated image may be displayed using the AR headset to form a graphical view as a superimposition on the patient's anatomy, as in block 966.

[0093] In a further configuration, the technique can be used to simulate a medical procedure. A patient's body or a patient's anatomical structure can be simulated using a simulated structure. The simulated structure can be plastic or cadaveric bone covered with soft materials (plastics and rubbers representing tissues, arteries, etc.) or other simulated materials. The simulated anatomical structure can include image visible markers or image visible markers with optical codes. The image dataset of the patient, on which a future procedure will be performed, can then be aligned with the simulated anatomical structure.

[0094] The actual medical tools used in the procedure may also be included in the simulation and may be within the field of view of the AR headset. Similarly, limited portions of the medical tools may be used (e.g., only the handles of the trocars) and the virtual tools may be viewed within the simulated patient. Additional overlays using real-time fluoroscopy, ultrasound, or other medical images captured in real-time may also be used. Thus, the medical professional may perform the same functions as described above as a simulation to better understand challenges, problems, or other issues that may arise in the actual procedure being planned. This simulation may also be used for training purposes.

[0095] The operations described with respect to this technique can be performed in any order that provides the described alignment results. For example, an X-ray generated image may first be aligned to the patient using image visible markers, and then the image dataset may be aligned to the X-ray generated image using shared anatomical structures (e.g., bones), which simultaneously aligns the image dataset to the patient as viewed through the AR headset.

[0096] 10 illustrates an exemplary system that may be employed to align an image dataset with a patient's body using image visible markers or image visible markers with optical codes as viewed through an AR headset. The system 1000 may include a sensor device 1002 (e.g., a camera, infrared light, LIDAR, other sensors), an augmented reality system 1020, a display device 1030, and multiple databases. The system 1000 may also include one or more processors, memories, file systems, communication units, operating systems, and user interfaces that may be communicatively coupled together. In some embodiments, the system 1000 may be, for example, a desktop computer, a server computer, a laptop computer, a smartphone, a tablet computer, an embedded computer, an AR headset, a VR headset, etc.

[0097] The sensor device 1002 may be configured to capture visible data, IR data, LIDAR data, and the like. In one example, the sensor device 1002 may be used to capture visible or IR data during a medical procedure. The visible or IR data captured by the sensor device 1002 may include images of a person's body (or a part of a body), one or more medical implements (e.g., medical instruments, implants, and the like), image visible markers or image visible markers with optical codes, and the like. The sensor device 1002 may transmit the captured imaging data to an augmented reality (AR) system 1020. The system may also include surface sensors, optical sensors, infrared sensors, Lidar sensors, or other sensors to detect and assist in the mapping of actual views or actual views detected by the AR system. Any object or surface may be detected for an operating room, a patient, a room, a physical shape, a medical implement, or any other physical surroundings or objects.

[0098] The augmented reality system 1020 may include an image processing engine 1022, an alignment module 1024, an image generation module 1026, and an extended display buffer 1028. For example, the image processing engine 1022 receives the captured visible or IR image data from the sensor device 1002 and analyzes the visible or IR image data to identify one or more light codes present in the light code database 1046, objects or people in the visible or IR image data. A number of different techniques may be used to identify objects or people in the visible image data, including, but not limited to, feature extraction, segmentation, edge detection, and / or object detection.

[0099] The image processing engine 1022 may also identify image visible markers or image visible markers having optical codes that may be affixed to the patient's body in the image or to medical implements in the visible or IR image data. Once the image processing engine 1022 identifies an optical code (e.g., AprilTag, barcode, QR code, etc.), the image processing unit 1022 may access the optical code database 1046 to retrieve information related to the optical code. In some examples, the optical code is associated with a particular patient, a particular procedure, or a particular object. The optical code may be used to more precisely identify the position and orientation of the body or fluoroscopy device.

[0100] In some embodiments, the registration module 1024 engages the image processing engine 1022 to register the image datasets relative to the person's body, the x-ray generated images, and / or each other. Once the position and orientation of the patient's body is determined, the registration controller 1026 can register any associated radiology images in the radiology image data 1042 with the patient's body. In some examples, the radiology images are received from the radiology image database 1042 based on the patient record in the patient record database 1040.

[0101] The image generation module 1026 can generate graphical data, virtual tools, 3D surgical paths, 3D coloring or shading of masses or organs, or highlighting of masses, organs or targets for display on a display device 1030 layered on top of the patient's body or medical tool. In some embodiments, this information can be loaded into an extended display buffer 1028. This information can then be transmitted to the display device 1030 (e.g., a waveguide or waveguide lens) of the AR headset for display to the user.

[0102] In one embodiment, the patient database 1040 includes a plurality of patient records. Each patient record may include one or more medical procedures performed on a patient. The patient records may also include notes, instructions, or plans for medical procedures. The patient records may also be associated with one or more radiology images in the radiology image database 1042. In another embodiment, the medical object data 1044 includes information describing medical items, including medical instruments, implants, and other objects.

[0103] In some configurations, the augmented reality system may be located on a server, which may be any computer system capable of functioning in conjunction with the AR headset or display device 1030. The server may be configured to communicate with the AR headset over a computer network to communicate image data to the AR headset or receive data from the AR headset.

[0104] 11 illustrates a computing device 1110 on which modules of the present technology may be implemented. A computing device 1110 on which a high level example of the present technology may be implemented is illustrated. The computing device 1110 may include one or more processors 1112 in communication with a memory device 1112. The computing device may include a local communication interface 1118 for components within the computing device. For example, the local communication interface may be a local data bus and / or any associated address or control bus as desired.

[0105] The memory device 1112 may include modules 1124 executable by the processor 1112 and data for the modules 1124. The modules 1124 may perform the functions described above. A data store 1122 may also be located within the memory device 1112 for storing an operating system executable by the processor 1112 as well as data related to the modules 1124 and other applications.

[0106] Other applications may also be stored in the memory device 1112 and executable by the processor 1112. The components or modules discussed herein may be implemented in software using a high-level programming language that is compiled, interpreted, or executed using a hybrid approach.

[0107] A computing device may also have access to I / O (input / output) devices 1114 that are usable by the computing device. One example of an I / O device is an available display screen for displaying output from the computing device. Other known I / O devices may be used with the computing device as desired. Networking devices 1116 and similar communication devices may be included in the computing device. This networking devices 1116 may be wired or wireless networking devices that connect to the Internet, a LAN, a WAN, or other computing networks.

[0108] The components or modules shown as stored in the memory device 1112 may be executed by the processor 1112. The term "executable" may refer to a program file in a format that can be executed by the processor 1112. For example, a program in a high-level language may be compiled into machine code in a format that can be loaded into a random access portion of the memory device 1112 and executed by the processor 1112, or source code may be loaded by another executable program and interpreted to generate instructions in a random access portion of the memory to be executed by the processor. The executable program may be stored in any portion or component of the memory device 1112. For example, the memory device 1112 may be a random access memory (RAM), a read-only memory (ROM), a flash memory, a solid state drive, a memory card, a hard drive, an optical disk, a floppy disk, a magnetic tape, or any other memory component.

[0109] The processor 1112 may represent multiple processors, and the memory 1112 may represent multiple memory units operating in parallel to the processing circuitry. This may provide parallel processing channels for processing and data within the system. The local interface 1118 may be used as a network to facilitate communication between any of the multiple processors and the multiple memories. The local interface 1118 may use additional systems designed to coordinate communications, such as load balancing, bulk data transfer, and similar systems.

[0110] Some of the functional units described herein have been labeled as modules to more particularly emphasize their implementation independence. For example, a module may be implemented as a hardware circuit comprising custom VLSI circuits or gate arrays, off-the-shelf semiconductors such as logic chips, transistors, or other discrete components. A module may also be implemented in a programmable hardware device such as a field programmable gate array, programmable array logic, programmable logic device, or the like.

[0111] Modules may also be implemented in software for execution by various types of processors. An identified module of executable code may include one or more blocks of computer instructions that may be organized as, for example, an object, a procedure, or a function. Nevertheless, the executable files of an identified module need not be physically located together, but may include disparate instructions stored in different locations that make up the module and that, when logically combined together, accomplish the purpose stated for the module.

[0112] Indeed, a module of executable code may be a single instruction, or many instructions, and may even be distributed across several different code segments, among different programs, and across several memory devices. Similarly, operational data may be identified and illustrated herein in modules, and may be embodied in any suitable form and organized within any suitable type of data structure. Operational data may be collected as a single data set or distributed across different locations, including across different storage devices. Modules may be passive or active, including agents operable to perform a desired function.

[0113] The techniques described herein may be stored on computer-readable storage media, including volatile and nonvolatile, removable and non-removable media implemented in any technology for storing information such as computer-readable instructions, data structures, program modules, or other data, including, but not limited to, RAM, ROM, EEPROM, flash memory or other memory technology, CD-ROM, digital versatile disks (DVDs) or other optical storage devices, magnetic cassettes, magnetic tapes, magnetic disk storage devices or other magnetic storage devices, or any other computer storage medium that may be used to store the desired information and the techniques described.

[0114] The devices described herein may also include communications or networking devices and networking connections that allow the devices to communicate with other devices. A communications connection is an example of a communication medium. Communication media typically embodies computer-readable instructions, data structures, program modules, and other data in a modulated data signal, such as a carrier wave or other transport mechanism, and includes any information delivery media. A "modulated data signal" means a signal that has one or more of its characteristics set or changed in such a manner as to encode information in the signal. By way of example, and not limitation, communication media include wired media, such as a wired network or direct-wired connection, and wireless media, such as acoustic, radio frequency, infrared, and other wireless media. As used herein, the term computer-readable media includes communication media.

[0115] Reference has been made to the examples illustrated in the drawings, and specific language has been used herein to describe the same. It will nevertheless be understood that no limitation of the scope of the technology is intended thereby. Alterations and further modifications of the features illustrated herein, as well as additional applications of the embodiments illustrated herein, that would occur to one of ordinary skill in the art in possession of this disclosure, should be considered within the scope of this specification.

[0116] Furthermore, the described features, structures, or characteristics may be combined in any suitable manner in one or more embodiments. In the preceding description, numerous specific details, such as embodiments of various configurations, have been provided to provide a thorough understanding of embodiments of the described technology. However, one skilled in the art will recognize that the technology can be practiced without one or more of the specific details, or with other methods, components, devices, etc. In other instances, well-known structures or operations have not been shown or described in detail to avoid obscuring aspects of the technology.

[0117] Although the present subject matter has been described in language specific to structural features and / or operations, it is to be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features and operations described above. Rather, the specific features and operations described above are disclosed as example forms of implementing the claims. Numerous modifications and alternative configurations can be devised without departing from the spirit and scope of the described technology.

Claims

1. 1. A method for registering an image dataset with a patient using an augmented reality (AR) headset, the method comprising: acquiring an image dataset and a two-dimensional (2D) X-ray generated image of an anatomical structure including at least one visible marker; registering the image dataset to the x-ray generated image by using data fitting to register the anatomical structures shared between the image dataset and the x-ray generated image; defining a position of the at least one visible marker in the image dataset using the x-ray generated image; and registering the image dataset with the patient based in part on the at least one visible marker correlated to the image dataset with reference to the at least one visible marker seen on the patient through the AR headset.

2. The method of claim 1 , further comprising displaying the image dataset as a superimposition on a patient's body visible through a lens of the AR headset.

3. 2. The method of claim 1 , wherein the at least one visible marker has an optical code having a known spatial relationship to at least one image visible marker, and wherein aligning the image dataset with the patient is based in part on the optical code of the at least one visible marker in the image dataset referenced to the optical code of the at least one visible marker as seen on the patient through the AR headset.

4. The method of claim 1 , wherein the visible markers are detectable on the patient using infrared light and an infrared sensor in the AR headset.

5. 1. A method for registering an image dataset with a patient's body, the image dataset being displayed using an augmented reality (AR) headset, the method comprising: acquiring an image dataset representative of a patient's anatomy; acquiring an x-ray produced image of at least a portion of the patient's anatomy in the image dataset and a radiopaque marker having an optical code; registering the image dataset to the x-ray generated image by using data fitting to register the anatomical structures shared between the image dataset and the x-ray generated image; determining a position of the radiopaque marker having the optical code in the image dataset using a registration with the x-ray generated image; and aligning the image dataset with the patient's body based in part on the optical codes of the radiopaque markers defined in the image dataset with reference to optical codes of the radiopaque markers visible on the patient through the AR headset.

6. 6. The method of claim 5, further comprising displaying the image dataset as a superimposition on the body of the patient visible through a lens of the AR headset.

7. 6. The method of claim 5, further comprising displaying the X-ray generated images using the AR headset to form a single graphical view as superimposed on the body of the patient visible through a lens of the AR headset.

8. The method of claim 5 , wherein the anatomical structure is at least one of bone, cartilage, skin, or an organ identifiable in the x-ray generated image.

9. 6. The method of claim 5, wherein the data fitting is at least one of a difference between images, a square of the difference, a best fit, a least squares best fit, a least squares best fit using the square of the difference, a curve fit, or a regression analysis.

10. The method of claim 5 further comprising acquiring at least two x-ray produced images to enable depth alignment using the radiopaque marker having the optical code.

11. 11. The method of claim 10, further comprising identifying a point of the radiopaque marker on the body of the patient by calculating a ray from an x-ray source that passes through a center of the radiopaque marker in each of the at least two x-ray generated images, and determining that the radiopaque marker is at an intersection of both rays.

12. 6. The method of claim 5, comprising identifying a point of the radiopaque marker on the body of the patient by calculating a ray from an x-ray source through a center of the radiopaque marker in an x-ray generated image, and determining that the radiopaque marker is at an intersection of the ray and the skin.

13. The method of claim 5 , wherein the optical code is formed into the metal of the radiopaque marker.

14. The method of claim 13 , wherein the radiopaque marker is made of metal and has an optical code affixed to the metal.

15. 6. The method of claim 5, wherein the optical code is at least one of a linear barcode, a matrix two-dimensional (2D) barcode, a quick response (QR) code, or an AprilCode.

16. identifying a plurality of optical codes on the X-ray imaging C-arm; determining an orientation of the X-ray imaging C-arm in 3D coordinate space relative to the AR headset or a patient from which an X-ray generated image is captured; 6. The method of claim 5, further comprising: constructing at least one of a slice, a slab, or a 3D volumetric projection of the image dataset in a plane aligned with the orientation of a plane of the X-ray generated image in the 3D coordinate space at a defined thickness visible through the AR headset.

17. The method of claim 16 , wherein the slices, slabs or 3D volumetric projections of the image dataset are reconstructed immediately after the x-ray generated images are captured of the patient.

18. 6. The method of claim 5, wherein the image dataset is a CT (Computed Tomography) image, an MRI (Magnetic Resonance Imaging), a 3D X-ray image, a Positron Emission Tomography (PET) image, a fluorescence modality, an Infrared Thermography (IRT) modality, a 3D mammography, or a Single Photon Emission Computed Tomography (SPECT) scan modality, a nuclear image, or an ultrasound image.

19. The method of claim 5 , wherein the image data set is a CT scan captured as at least one of a maximum intensity projection (MIP) image, a mean intensity image, or a minimum intensity image.

20. The method of claim 5 , wherein the X-ray generated image has a defined position and orientation relative to the AR headset or the body of the patient.

21. 1. A method for aligning an image dataset representative of medical imaging of a patient with a body of the patient using an augmented reality (AR) headset worn by a medical professional, the method comprising: acquiring an image dataset representative of the patient's anatomy; acquiring an x-ray generated image of at least a portion of the anatomical structure within the image dataset; registering the image dataset to the x-ray generated image by using data fitting to register the anatomical structures shared in the image dataset and the x-ray generated image; determining a position of a radiopaque marker having an optical code in the image data set using a registration with the x-ray generated image; registering the image dataset with the patient's body using optical codes of radiopaque markers visible on the patient and the optical codes of the radiopaque markers defined in the image dataset; determining an orientation and position of an X-ray imaging C-arm using a plurality of optical codes on the X-ray imaging C-arm by which the X-ray generated image was captured; constructing a slice of the image data set within a plane based in part on an orientation and position of the x-ray generated image plane, the slice having a defined thickness; and displaying the slices and the X-ray generated images to form a graphical view as superimposed on the patient using the AR headset.

22. 22. The method of claim 21, wherein the image data set is a CT scan captured as at least one of a maximum intensity projection (MIP) image, a mean intensity image, or a minimum intensity image.

23. 22. The method of claim 21, wherein the shared anatomical structure is at least one of bone, cartilage, skin, or organs visible in the x-ray generated image.

24. 22. The method of claim 21, wherein the data fitting is at least one of a least squares best fit, a curve fit, or a regression analysis using the squares of the differences.

25. 22. The method of claim 21, further comprising using at least two x-ray generated images to enable registration of the x-ray generated images with the image data set using the radiopaque markers.

26. 22. The method of claim 21, wherein the optical code is made of metal and has an optical code formed in the metal.

27. 22. The method of claim 21, wherein the optical code is made of metal and has the optical code attached to the metal.

28. 22. The method of claim 21, wherein the optical code is at least one of a linear barcode, a matrix two-dimensional (2D) barcode, a quick response (QR) code, or an AprilCode.

29. 1. A system for registering medical imaging of a patient with a body of the patient, the medical imaging being displayed using an augmented reality (AR) headset worn by a medical professional; At least one processor of the AR headset; a memory device of the AR headset, when executed by the at least one processor, for providing the system with: acquiring an image data set and a two-dimensional (2D) x-ray generated image of an anatomical structure including at least one image visible marker; registering the image dataset to the x-ray generated image by using data fitting to register the anatomical structures shared between the image dataset and the x-ray generated image; determining a position of an image visible marker having an optical code in the image data set using a registration with the x-ray generated image; and a memory device containing instructions for aligning the image dataset with the patient's body as visible through the AR headset based in part on the image visible markers having the optical codes visible on the patient with reference to optical codes of the image visible markers represented in the image dataset.

30. 30. The system of claim 29, further comprising displaying the image dataset using the AR headset to form a superimposition on the body of the patient.

31. 30. The system of claim 29, wherein the data fitting is at least one of a least squares best fit, a curve fit, or a regression analysis using the squares of the differences.

32. 30. The system of claim 29, further comprising acquiring at least two x-ray produced images to enable alignment using the image visible markers.

Citation Information

Patent Citations

  • Systems and Methods for Fusion Image-Based Guidance with Late Marker Placement

    JP2018514352A

  • Image-guided surgery with surface reconstruction and augmented reality visualization

    US20160191887A1

  • Method for patient registration, calibration, and real-time augmented reality image display during surgery

    US20190192230A1

  • Aligning image data of a patient with actual views of the patient using an optical code affixed to the patient

    US20190348169A1

  • Mirroring in Image Guided Surgery

    US20210186647A1