Registration of medical images in augmented reality displays

Abstract

An AR headset for coexistence of an image dataset with a body of a person is described.SOLUTION: One method may include using the AR headset to identify an optical code 200 having a contrast agent in a tube on a person's body. The image data set can be aligned with the body of the person using the fixed position of the image visible marker with the contrast agent in the tube relative to the optical code that references the representation of the image visible marker 206. In one configuration, the image data set 216 may be scaled by comparing the measured size of the image visible marker (e.g., gadolinium tube) in the captured image data set to the known size of the image visible marker. Further, the center of the optical code may be identified to more accurately align the image data set with the body of the person.SELECTED DRAWING: Figure 2

Description

[Technical Field]

[0001] The present disclosure relates to the registration of medical images in an augmented reality display. [Background technology]

[0002] Mixed reality or augmented reality is an area of ​​computing technology in which images from the physical world and virtual computing worlds can be combined into a mixed reality world. In mixed reality, people, places, and objects from the physical and virtual worlds become a blended environment. Mixed reality experiences may be delivered through existing commercial or custom software, along with the use of virtual reality (VR) or augmented reality (AR) headsets.

[0003] Augmented reality (AR) is an example of mixed reality in which a live, direct or indirect view of a physical, real-world environment is augmented or supplemented with computer-generated sensory input, such as images, audio, video, graphics, or other data. Such augmentation can be achieved when real-world locations are viewed and associated with environmental elements. With the aid of advanced AR technologies (e.g., adding computer vision and object recognition), information about the real world around the user can be made interactive and digitally modified (e.g., via computer graphic overlays).

[0004] A problem faced by AR systems or AR headsets is identifying the position and orientation of objects with high accuracy. Similarly, aligning the position of virtual elements with a live view of the real-world environment can be difficult. For example, aligning a virtual object with a physical object when viewed through an AR headset can be difficult because the luminescence or brightness of the virtual object obscures the underlying real-world object with which the virtual object is to be aligned. While aligning a virtual object with a physical object to within approximately a few centimeters may be useful for entertainment and less demanding applications, higher positioning and alignment resolution for AR systems may be desirable in scientific, engineering, and medical fields. As a result, the positioning and alignment process for AR used in scientific, engineering, and medical fields may be performed manually, which can be time-consuming, cumbersome, and inaccurate. [Brief explanation of the drawings]

[0005] [Figure 1] An exemplary augmented reality (AR) environment is shown in which a patient's image dataset can be referenced and aligned to the patient's actual view using one or more visible optical codes fixed to contrast media or contrast markers in tubes attached to the visible optical codes. [Figure 2] 1 shows an example of an optical code with image visible markers, which is a tube attached to the optical code and the patient. [Figure 3] An example is shown using multiple segments of a tube as image visible markers. [Figure 4a] 1 is a flow chart illustrating a method for manufacturing an encapsulated contrast agent or contrast marker in a tube that is affixed to an optical cord. [Figure 4b] 10 is a flow chart illustrating another sequence of a method for manufacturing an encapsulated contrast agent or contrast marker in a tube affixed to an optical cord. [Figure 5]10 shows an example view of an AprilTag with an attached tube containing contrast agent or contrast markers. [Figure 6A] 1 shows an example of a tube containing a contrast agent or contrast fluid when captured using MRI (magnetic resonance imaging). [Figure 6B] 1 shows examples of different diameters of tubes containing contrast agents when captured using MRI (Magnetic Resonance Imaging). [Figure 7] 1 illustrates scaling of an image dataset by comparing the measured size of an image-visible marker (e.g., gadolinium fluid in a tube) in a captured image dataset with a known or measured size of the image-visible marker. [Figure 8] We demonstrate locating and using the center of an optical code for alignment or co-localization of an image dataset with a human body using an augmented reality (AR) headset. [Figure 9a] 1 is a flowchart of an exemplary method using an augmented reality (AR) headset to co-locate an image dataset with a human body using image visible markers that are tubes containing a contrast or marking agent. [Figure 9b] 1 illustrates a method for scaling an image dataset registered to a human body by comparing measured sizes of geometric attributes of image visible markers in the image dataset with known sizes of the geometric attributes of the image visible markers. [Figure 9c] 1 illustrates a method for registering an image dataset to a human body by finding the center of an optical code. [Figure 10] 1 illustrates an exemplary system that can be employed to register an image dataset to a human body. [Figure 11] FIG. 1 is a block diagram illustrating an example of a computing system for processing the present technology. DETAILED DESCRIPTION OF THE INVENTION

[0006] Techniques are provided for using an augmented reality (AR) headset to identify one or more optical codes (e.g., AprilTag, QR Code, or 2D optical barcode) within a view of a camera of the AR headset during a medical procedure. An image dataset can be registered with a person's body using one or more optical codes and image-visible markers on the person's body. The image dataset can be a previously acquired image of a portion of the person's body using a non-optical imaging modality. Examples of non-optical imaging modalities can be an MRI (magnetic resonance imaging) modality, a CT (computed tomography) scanning modality, an X-ray modality, a Positron Emission Tomography (PET) modality, an ultrasound modality, a fluorescence modality, an Infrared Thermography (IRT) modality, a 3D mammography, or a Single-Photon Emission Computed Tomography (SPECT) scanning modality, etc. The image dataset can be registered to the person's body using image visible markers that are a fixed distance from at least one optical code positioned on the person's body (e.g., on the side of the body or under the body). For example, the image visible markers and the optical code can both be mounted on a piece of material (e.g., located in the same location or fixedly adjacent to each other) to facilitate registration of the image dataset with the person's body. The image visible markers are markers that can be seen in a non-visible imaging modality, such as a captured radiology image or image dataset, that may not be optically visible to the AR headset. The image dataset can be captured with representations of the image visible markers in machine-captured images that capture the structure of the human body using a non-optical imaging modality. The representations of the image visible markers in the image dataset can be registered with the patient's body using known fixed positions of the image visible markers relative to one or more optical codes affixed to the person's body (as described in more detail below).For example, the image-visible marker can be an enclosed container (e.g., an MRI bead or sphere) or an MRI contrast agent within a length of tubing. For other imaging modalities, such as those mentioned above, a suitable contrast agent can be provided within a tubing that is visible in images captured in the respective modality (such as nuclear-based image acquisition).

[0007] FIG. 1 illustrates an example of an augmented reality (AR) environment 100 in which an image dataset 116 of a patient or person 106 can be aligned with an actual view of the patient 106 using an optical code 200 affixed to the patient 106. The image dataset 116 can include imaging of a tube 198 or sealed container containing an image-visible fluid, image-visible medium, or image-visible marker within the image dataset. The identity (e.g., a unique optical code) of a distinct optical tag 200 within the image dataset 116 can be determined or encoded based on the location of the tube 198 containing the image-visible medium or marker and the length of the tube. The optical code 200 can also be identified within a camera image captured by the AR headset 108 using a visible AprilTag or 2D barcode as the optical code. Thus, the image-visible marker 198 can be identified within the image dataset 116 and aligned with the optical code 200 visible through the AR headset 108.

[0008] The environment 100 may include a physical space 102 (e.g., an operating room, a laboratory, etc.), a user 104, a patient 106, a plurality of optical codes 200 on the patient, a medical tool 118 having an optical code 198, and an AR headset 108 in communication with a server 112 via a computer network 110. A virtual user interface 114 and a virtual cursor 122 are shown in dashed lines to indicate that these virtual elements are generated by the AR headset 108 and are viewable by the user 104 through the AR headset 108.

[0009] The AR headset 108 may be an AR computing system that can augment a real-world view of the patient 106 (covered by hospital gown or fabric 107) with image datasets 116. For example, the AR headset 108 may be used by the user 104 to augment a real-world view of the patient 106 with a view of one or more 3D image datasets or radiology images of the patient 106, including, but not limited to, bones (as illustrated in FIG. 1 ), muscles, organs, or fluids 106b.

[0010] The AR headset 108 may enable the image dataset 116 (or a projection of the image dataset) to be dynamically reconstructed. Thus, as the user 104 moves around the patient 106, sensors in the AR headset 108 determine the user 104's location relative to the patient 106, and the patient's internal anatomical structures displayed using the image dataset can be dynamically reconstructed as the user 104 selects different orientations relative to the patient. For example, the user 104 may walk around the patient 106. The AR headset 108 can then augment the actual view of the patient 106 with one or more acquired radiology images or image datasets (MRI, CT scan, etc.) of the patient 106, so that both the patient 106a and the image dataset 116 of that patient 106a can be viewed by the user 104 from any angle (e.g., projected images or slices from the image dataset can also be displayed). The AR headset 108 may be a modified version of the Microsoft HOLOLENS®, Meta Company META 2, Epson MOVERIO®, Garmin VARIA VISION, or other AR headset.

[0011] The optical code 200 may be affixed to the patient 106 prior to generation of image data of the patient 106 (e.g., acquisition of an MRI, CT scan, X-ray, etc.), and then remain affixed to the patient 106 while the patient 106 is viewed by the user 104 via the AR headset 108. The optical code 200 and image visible markers 198 may then be used by the AR headset 108 to programmatically align an image dataset 116 of the patient 106 with an actual view of the patient 106. For example, one or more optical codes 200 associated with the body of the person 106 may be identified. The optical code 198 on the person's body 106 a may be positioned in a fixed position relative to the image visible markers 198, which may include contrast agent within a tube, or a length of tube. The image visible markers 198, which may use contrast agent within a length of tube, may be identified in an image dataset 116 acquired using a medical imaging device. The image dataset 116 can be aligned with the person's body 106b when viewed through the AR headset 108 using one or more optical codes 200 on the person's body and fixed positions of the image visible markers 198 with contrast agent in tubes relative to the optical codes 200. The representation of the image visible markers 198 with contrast agent in tubes in the image dataset can be used as reference points for the alignment process.

[0012] The contrast agent or agent may be an MRI (magnetic resonance imaging) visible contrast agent. In one example, the encapsulated contrast agent or agent in a sealed container or tube may be a paramagnetic fluid or semi-fluid. The contrast agent may be encapsulated in a sealed container or tube of a defined length or a variable length selected by a person. The defined length of the tube may be associated with or identified as an optical code in a 2D data encoding that is visible using an optical sensor in an AR headset. The defined length of the tube may be associated with an optical code that is visible by a camera or optical sensor in an AR headset by linking the predetermined length of the tube with the optical code in a data store or database. This registration or linking between the visible optical code and the tube may be accomplished by capturing an image of the optical code with the tube at a fixed distance from the optical code and recording the measured length of the captured tube, or by manual entry.

[0013] The segments of the tube can vary (e.g., 3 mm, 5 mm, 7 mm, 10 mm), and specific lengths of tube can be associated with distinct optical codes. For example, optical code A can have a tube attached to an optical code that is 5 millimeters (mm) long, optical code B can have a tube attached to an optical code that is 7 mm long, and optical code C can have a tube attached to an optical code that is 10 mm long. The segments of the tube can be linked to their respective optical codes. Thus, when an image visible marker 198 created by a tube having a particular length or enclosure is identified in the image dataset 116, the corresponding optical code and the location of the optical code are also known to the image dataset.

[0014] The corresponding optical code 200 is known to be a fixed distance and orientation from the image visible marker with the tube or sealed container. Because each individual tube segment may be different, the length of the tube section can be used to align the image data with the patient's visible body. Matching the image visible marker made of the tube with the representation of the tube in the image dataset and / or optical code can be performed using the tube length, because each length or combination of multiple tube segments may be unique. As a result, the tube length cannot be matched to an incorrect optical code, as the data store of links between tube segments and optical codes will be checked to prevent mismatches. Furthermore, if a 5 mm tube representation in the image dataset is attempted to match a 10 mm tube that is visible or expected for the optical code, this will cause a mismatch, and the system will attempt to match the correct tube length to the correct tube representation.

[0015] In one example, different sizes, shapes, or lengths of tube 198 can be positioned in fixed locations according to the optical code 200. Each optical code (e.g., AprilTag or 2D barcode) has a unique optical code 200, but the optical code is not visible in the image dataset. The tubes that are visible in the image dataset identify 1) where the visible optical code is and 2) which visible optical code is associated with the image visible code. Once the location of the tube is known, the location of the visible optical code 200 is also known. Finding the location of the optical code 200 and the tube allows the system to stitch the image dataset (e.g., MR images) with a real or visual view of the person's body via an A / R headset.

[0016] The tube containing the contrast agent or imaging agent can be a shaped tube in the shape of a line, a cross formation, concentric circles, a square, a rectangle, a right angle, or a lattice structure. The above shapes can be created in a single plane with other tube segments of that shape (although the tube has a three-dimensional depth due to the diameter of the tube). The tube can have a cross-sectional shape that is at least one of a circle, a triangle, a square, a rectangle, an ellipse, a hexagon, an octagon, an oval, a semicircle, or a polygon. The encapsulated contrast agent or imaging agent can be a gadolinium solution (e.g., a gadolinium chelate), copper sulfate, an iron solution, manganese, vitamin E oil, or other magnetic resonance imaging medium. The tube can be a plastic tube, a glass tube, a composite tube (e.g., glass and epoxy), or another non-metallic tube. The shape of the image-visible marker can be used to determine which unique optical code or visual marker is at the location of the unique image-visible marker.

[0017] Additionally, the same optical code 200 used during capture of the image dataset 116 can be used to automatically acquire the image dataset to ensure that the image dataset retrieved by the AR headset 108 matches the actual patient 106 viewed through the AR headset 108. Similarly, the length or shape of the tube 198 can also be checked against the image dataset to validate that the correct patient procedure is being performed (e.g., checking the tube pattern is correct for the patient). If the patient has a tube pattern that does not match the image dataset (or vice versa), an error can be flagged.

[0018] The virtual user interface 114 generated by the AR headset 108 can include options for changing the display of the projected internal anatomical structures of the patient 106 from the image dataset 116 of the patient 106. The virtual user interface 114 can include other information that may be useful to the user 104. For example, the virtual user interface 114 can include information about the patient or medical implement 118 (e.g., medical instruments, implants, etc.) identified with an optical code. In another example, the virtual user interface 114 can include a medical chart or other medical data of the patient 106. In some configurations, the person's image data 116 or captured radiological data can be displayed by the AR headset 108 using a large amount of the image dataset 116 to display the patient's 106a's radiologically captured anatomical structures (e.g., bones 106b, tissue, blood vessels, fluid, etc.) from the image data. The image data can include axial, coronal, sagittal, or oblique slices of the image data. The slices may be two-dimensional (2D) slices having depth as well as height and width (e.g., one or more layers of voxels), three-dimensional (3D) slices, and / or four-dimensional (4D) slices (3D images with a time series of images). The user 104 can control the virtual user interface 114 using hand gestures, voice commands, eye movements, a remote control (e.g., a finger clicker), a 3D mouse, a VR wand, a finger sensor, haptic technology, or other control methods.

[0019] FIG. 2 shows the optical code 200 of FIG. 1 affixed to the patient 106 of FIG. 1 . Referring to both FIGS. 1 and 2 , the optical code 200 may be perceptible to an optical sensor, such as an optical sensor for the visual spectrum or a camera, built into the AR headset 108. In some embodiments, the optical code 200 may be an AprilTag, a linear barcode, a matrix two-dimensional (2D) barcode, a Quick Response (QR) code, or some combination thereof. AprilTag is a two-dimensional barcode that can be a visual reference system useful for augmented reality and camera calibration. AprilTags can be used to calculate the tag's 3D position, orientation, and identity relative to a camera, sensor, or AR headset.

[0020] The optical code 200 can be further associated with a marker 206, 214, or image-visible marker, that is perceptible to a non-optical imaging modality. An example of a non-optical imaging modality used to capture an image-visible marker using a tube containing a paramagnetic fluid can be an MRI modality. In another example, the non-optical image or image dataset can be an image or image dataset that includes a combination of two or more forms of non-optical imaging modalities (e.g., two or more images combined together, combined segments of two or more non-optical images, a CT image fused with an MRI image, etc.). Each image dataset in a separate modality can have an image-visible code within the individual image dataset, allowing PET images, CT images, MRI images, fluoroscopic images, etc., to be registered and referenced with the optical code on the person's body within the AR system view. Forming the marker 206 from a material that is perceptible to a non-optical imaging modality can enable the marker 206 or image-visible marker to appear in an image dataset of the patient 106 captured using any combination of non-optical imaging modalities.

[0021] The marker 206 or image-visible marker can comprise one or more segments of tubing containing paramagnetic fluid 206. The tubing segments can be provided separately, as in FIG. 2, or can be arranged in a pattern and have a fixed position relative to the location of the optical code 200. Multiple markers 214 or multiple sections of tubing with paramagnetic material can optionally be used to form a recognizable pattern that identifies the location and / or image-visible marker. For example, in the embodiment disclosed in FIG. 2, the optical code 200 can be printed on a material 202 (such as an adhesive bandage, paper, plastic, etc.), and the markers 206 can be affixed to (e.g., attached to or adhesively attached to) the material 202. In this embodiment, the one or more markers 206 can be positioned in a fixed position relative to the location of the optical code 200 by being positioned in a fixed position relative to the bandage 202 or material.

[0022] Additionally, the markers 206 can be placed by (at least temporarily) attaching or printing the optical code 200 directly onto the skin 106a of the patient 106. By placing the markers 206 in a pattern on the skin of the patient 106 that has a fixed location relative to the location of the optical code 200, this fixed location can be used to calculate the location of the markers 206 or pattern of image visible markers relative to the visible location of the optical code 200. The visibility of the markers 206 or tubes to the sensors of the AR headset 108 can then also be used for alignment.

[0023] Once the optical code 200 and markers 206 are affixed to the patient 106 in a fixed pattern, a non-optical imaging modality (in which the markers 206 are perceptible) can be used to capture an image dataset 216 of the patient 106 and markers 206 (only a partial segment shown). In particular, the image data can include the internal anatomical structures of the patient 106 (such as bones 106b, muscles, organs, or fluids), as well as a pattern of markers 206, 214 in fixed positions relative to the positions of the internal anatomical structures of the patient 106. In other words, not only will the internal anatomical structures of the patient 106 appear in the image data of the patient 106, but the markers 206 will also appear in the image dataset of the patient 106 in a pattern, and the positions of this pattern of markers 206 will appear in the image dataset in fixed positions relative to the positions of the internal anatomical structures of the patient 106. In one example, when the non-optical imaging modality is an MRI modality, the MRI image may display the bones, organs, and soft tissues of the patient 106, as well as a marker 206 having a tube positioned in a fixed position relative to the position of the bones, organs, and soft tissues of the patient 106.

[0024] Furthermore, the patient 106 may be moved, for example, from a hospital medical imaging suite to a hospital operating room. The user 104 (such as the medical professional in FIG. 1 ) can then use the AR headset 108 to determine the location of the optical code 200 (and, optionally, the visible portion of the markers 206, if visible) on the person's or patient's body. The AR headset 108 can then automatically retrieve image data of the patient 106 based on the optical code.

[0025] After detecting the optical code 200 in the 3D space 102, the AR headset 108 can automatically calculate the location of one or more markers 206 with respect to each other using the tube in the 3D space 102. This automatic calculation can be based on the detected position of the optical code 200 in the 3D space 102, the known fixed position of the pattern of markers 206 on the tube relative to the position of the optical code 200, or visually viewing a segment of the tube filled with paramagnetic fluid. Even if the markers 206 are not perceptible to the AR headset 108 (e.g., because the markers 206 are clear plastic or hidden under the optical code printing material), the AR headset 108 can automatically calculate the location of the pattern of markers 206 based on the position of the optical code 200 and the fixed position of the pattern of markers 206 relative to the position of the optical code 200. In this example, these fixed positions may allow the AR headset 108 to automatically calculate the position of the patterns of markers 206 in 3D space 102 relative to each other, even though the AR headset 108 does not directly sense the positions of the markers 206.

[0026] After calculating the location of the markers 206 or the pattern of image-visible markers in the 3D space 102, the AR headset 108 can register the position of the internal anatomical structures of the patient 106 in the 3D space 102 by aligning the calculated position of the pattern of markers 206 in the 3D space 102 with the position of the pattern of markers 206 in the image dataset. This alignment can be performed based on the calculated position of the pattern of markers 206 of the tube in the 3D space 102 and the fixed position of the image dataset relative to the markers 206. This alignment and registration can then enable the AR headset 108 to display the internal anatomical structures of the patient 106 in real time from image data projected onto an actual view of the patient 106. Thus, the optical code 200 and one or more associated markers 206 can be used by the AR headset 108 to automatically align the image data of the patient 106 with the actual view of the patient 106.

[0027] 3 illustrates that multiple optical codes 312, 313 may be affixed to the patient 106 simultaneously to further ensure accurate alignment between the image dataset of the patient 106 and the actual view of the patient 106 in 3D space 102. Additionally, the pattern of markers 314, 312 of each optical code 310 may use multiple segments of tubing. Furthermore, because the markers 314 are affixed to the outer layer of the patient 106, the markers 314, 312 do not necessarily all lie in a single actual plane, but rather may conform to any curvature of the outer layer of the patient 106. In these embodiments, the position of the pattern of markers 314, 312 relative to the position of the optical code 314 is established after affixing the optical code 310 and markers 314, 312 to the patient 106 to account for any curvature on the outer layer of the patient 106.

[0028] FIG. 3 further illustrates a visual image that may be captured by the camera of an AR headset or system. The visual image may include a patient's 106 body 300 having an optical code 310 printed on a material 304 affixed to the body 300. A first marker 314 is shown oriented substantially perpendicular to a second marker 312. The first marker 314 and the second marker 312 may be tubes containing paramagnetic fluid. The use of the second marker 312 or the third marker 316 may be optional and is illustrated using dashed lines. Using multiple orientations for the markers may be useful because multiple markers can help properly orient the image dataset and avoid flipped or inverted images. FIG. 3 also illustrates an image visible marker 320 having a tube on the second optical code 313. This image visible marker may be oriented in any orientation relative to the optical code. The orientation of the markers 320 can be parallel to the edge of the optical code, perpendicular to the edge of the optical code, or oblique to the edge of the optical code. Additionally, as previously discussed, optional radiopaque markers 302 can be included with the optical code for use in image registration with imaging modalities other than MRI.

[0029] In another configuration, the tube can include multiple tube sections, each having a distinct length at a fixed distance and orientation relative to the optical code 200. For example, each optical code 200 can have two or three sections of tube attached to the optical code 200. For example, three different segments of tube can be oriented in the same way, with the combination of the tube segments being unique and also identifying a unique optical code. Alternatively, each section of tube can extend from a distinct corner or vertex of the optical code 302.

[0030] The use of a tube with a paramagnetic fluid as an image-visible marker can aid in alignment in an MRI (magnetic resonance imaging) environment, as the use of iron or metallic materials in MRI images tends to cause artifacts in the MRI images. More specifically, the use of iron or metallic materials in MR imaging causes attenuation, loss, or destruction of the signal generated by the protons of water molecules when iron or metallic materials are used in MRI imaging. Therefore, the present technology can use a marker in an MRI image that is a polyethylene tube filled with a paramagnetic material, such as gadolinium fluid.

[0031] 4a is a flow chart illustrating a method for manufacturing an encapsulated contrast agent or contrast marker in a tube. One operation can be sealing a first end of a plastic tube, as in block 402. The plastic tube can be sealed using a heated wire to cut and seal the first end of the tube. Alternatively, a heated knife, a heat crimping tool, or another heated sealing tool can be used to seal the first end of the plastic tube.

[0032] Another operation is to cut a length of plastic tubing to a defined or measured length that can be captured using a medical imaging device, as in block 404. The length of the plastic tubing can be cut to any length that can be captured by the medical imaging device. For example, the length of the plastic tubing can be sized from just a few millimeters up to several centimeters, as may be useful for the imaging scenario. The size of the plastic tubing is preferably less than the size of the imaging window that the medical imaging device (e.g., an MRI device) is expected to capture.

[0033] As per block 406, the plastic tube can be filled with a contrast agent or contrast marker that is visible in the MRI image. As previously mentioned, the contrast agent can be a paramagnetic fluid, such as gadolinium fluid, or another contrast agent that is visible to the MRI imaging device. As per block 408, the second end of the plastic tube containing the imaging contrast agent can be sealed. The material within the tube can include a paramagnetic material and a fluid, and the paramagnetic fluid will appear as a bright area in the 3D MRI image. In other words, the contents of the tube will be visible in the MRI image.

[0034] As in block 410, one or more optical codes can be attached to the plastic tube using a bonding or mounting structure to maintain a fixed position between the plastic tube and the one or more optical codes. This bonding or mounting structure can be paper or plastic on which the optical codes are printed, or the bonding structure can be any paper, plastic, or non-metallic structure used to fix the distance and orientation between the optical codes and the tube. Thus, the tube can be used to identify the location of the optical code or encode the position of a unique optical code by using the size (e.g., length or diameter), shape, or other geometry of the tube.

[0035] 4b shows that the second end of the plastic tube can be sealed 408 before the fluid is added to the plastic tube. As in block 406, in this sequence of the method, the plastic tube can be filled with a medical contrast agent visible using MRI after the plastic tube is sealed. In this situation, the contrast agent can be injected into the plastic tube and the injection hole can be sealed using heat or an adhesive material covering the injection hole. For example, a dilute paramagnetic solution such as gadolinium, copper sulfate, or iron solution can be filled through one end of the tube before the tube is closed.

[0036] The tubes illustrated in Figures 4a and 4b can also be sealed using other sealing agents, examples of which may include sealing the ends of the tube with adhesive, using clay filler plugs, applying an adhesive plastic layer over the ends, sticking plastic end caps to the ends of the tube, adhering metal foil to the ends, adhering a metallized plastic sheet to the tube, or using any other sealing method or material.

[0037] FIG. 5 illustrates an optical code 502 and a tube 504 containing image-visible markers or media. The optical code can be an AprilTag (as shown), a 2D barcode, or other optical code that can be captured using a camera. The length of the tube 504 can have a length similar to the length of the side or edge of the optical code 502. However, the size of the length of the tube can be shorter or longer than the length of the edge of the optical code 520. Furthermore, while the length of the tube is shown as being substantially parallel to the side of the optical code, the length of the tube can be set at any angle relative to the optical code 520. For example, the tube can be at a 10-degree angle relative to the optical code 520 or perpendicular to the side of the optical code. The tube can also be a bendable plastic, which can deform and allow the tube to follow the contours of the patient's body or the anatomical structure being imaged. Even if the tube is curved, a medical imaging tool can measure the path of the tube in 3D and determine the length of the tube.

[0038] Figure 6A shows tubes 610, 612 containing parametric fluid as captured by an MRI imaging device. The tubes in Figure 6a are captured above a container of contrast agent or parametric fluid. Figure 6B shows multiple segments of tube 620A-C captured in an MRI image dataset. The multiple tube segments are visible from one end of the tube. For reference, a reference container containing parametric fluid is below the tube segments in Figure 6B.

[0039] As discussed above, the ability to register an image dataset with anatomical structures within a patient's body is extremely helpful when lining up medical instruments and performing other tasks in a medical procedure. If an image dataset does not line up near previously imaged anatomical structures or actual human tissue, the image dataset will be of little use in the context of a medical procedure. For example, if an image dataset representing human tissue (e.g., an imaged hole in the skull) does not line up with actual human tissue (e.g., the actual hole in the skull), the image dataset will be of little use in the context of a medical procedure. In some cases, a coarsely registered image set may be useful for simulation or training purposes, but not for surgical procedures. In contrast, the present technology enables image datasets for MRI images with a resolution that can enable medical personnel to use the image dataset in medical procedures with high precision.

[0040] In medical image acquisition using a non-optical imaging modality, the number of voxels in an image may be, for example, 512 x 512 voxels. Each voxel in the image may represent a predetermined number of millimeters or sampling frequency at which the image dataset was captured from the patient's body. For example, this sampling frequency may be expected to be 1.2 millimeters per voxel (Note: This is an arbitrary example measurement; the actual sampling frequency may vary). However, if the medical image scanner is slightly miscalibrated and the voxels in the image dataset are actually sampled at 1.15 millimeters per voxel, the actual size of the human anatomical structure or real human tissue being imaged will differ from what the image dataset or image scanner can capture (e.g., the image dataset may be too small). Such miscalibration may result in the virtual human tissue being slightly larger or smaller than the real human tissue, which is a result of the inaccurate sampling.

[0041] As a result, the present technology can use image visible markers to scale the image dataset displayed to the user of the AR headset. More accurate scaling can help reduce image distortion and correct other errors. The image visible markers can be identified within an image dataset that includes a captured or electronic version of the image visible marker. The actual size of the image visible markers may be measured before the medical procedure, or the actual size of the image visible markers may be known in advance at the time of manufacture. Alternatively, the size of the image visible markers can be measured from a visible light image (i.e., a photograph or video) captured by the camera of the AR headset. For example, the size of the image visible markers can be measured in the image captured by the camera using a reference object (e.g., an optical code or medical instrument) or other means.

[0042] A comparison can be made between the known size of the image visible marker in the real world and the measured size of the image visible marker data captured in the image dataset. This comparison can determine whether an error or size difference exists between the electronic representation of the image visible marker in the image dataset and the known size of the image visible marker. If a difference is identified, the image dataset can be scaled to correct the difference or error. Correcting scaling issues in the image dataset results in the image dataset being better suited for use in medical procedures. This technique can correct any erroneous image scaling by increasing or decreasing the image scaling. For example, if the image dataset size is off by 5% and a medical procedure is performed in which a needle travels 20 cm through human tissue, the tip of the needle will miss its intended location in the human tissue for the procedure by 1 cm. Missing the desired location by 1 cm in a procedure is a significant amount in the human body and can have serious consequences for individual patients. Correctly scaling and accurately registering image data can reduce such problems in medical procedures.

[0043] FIG. 7 illustrates this scaling of the image dataset 720 by comparing the measured size of the image visible marker 714 (e.g., a gadolinium tube) in the captured image dataset 720 to the known size of the image visible marker 712 or the measured size of the image visible marker.

[0044] The known size of the geometric attribute of the image visible marker can be identified. The geometric attribute of the optical code can be the length of a tube containing a parametric fluid, the distance between two metal markers 708 attached to the optical code 704, the radius of a circle encompassing the multiple metal markers of the optical code, the diameter of the image visible marker, or another geometric measurement of the image visible marker. The known size of the geometric attribute can be stored by a medical technician using a physical measuring device, or can be retrieved from a data store in which the known size was set at manufacturing time. For example, the size of the geometric attribute of the optical code can be a pre-measured value stored in a data store accessible to the AR headset. As discussed above, the known size of the geometric attribute of the image visible marker can be measured using a visual image or photograph captured of the optical code and the image visible marker using the visible light camera of the AR headset. This assumes that the image visible mark can be seen using the AR headset camera, which is not always accurate.

[0045] The measured size of the geometric attribute of the image visible marker in the image dataset can be compared to the known size of the geometric attribute of the image visible marker to determine a calculated difference in size between the measured size of the geometric attribute of the image visible marker in the image dataset and the known size of the geometric attribute of the image visible marker.

[0046] The scaling of the image dataset can be modified based in part on the calculated difference in size between the known size and the measured size of the geometric attributes of the image visible markers. This scaling can allow the image dataset to be more accurately aligned with the person's body when viewed through the AR headset and the image visible markers using one or more optical codes on the person's body. The scaling modification can increase or decrease the magnification of the image dataset to match the measured size of the geometric attributes of the image visible markers in the image dataset to the known size of the geometric attributes of the image visible markers.

[0047] More specifically, the scaling of the image dataset when viewed through the AR headset can be increased if the known size of the geometric attributes of the optical code is greater than the measured size of the image visible markers in the image dataset. Conversely, the scaling of the image dataset when viewed through the AR headset can be decreased if the known size of the geometric attributes of the optical code is less than the measured size of the image visible markers in the image dataset.

[0048] This technique allows for a calculated calibration between the known size of an object and the measured size of the object in the image dataset during a medical procedure, without necessarily requiring intervention from medical personnel. For example, the known length of the image-visible marker may be 10 millimeters. The image-visible marker can be found in the MR image, and the scale of the image-visible marker can be determined using an electronic tool in the imaging software. The system can then compare the size of the image-visible marker to the known size of the object in the image dataset. If the image-visible marker is too small or too large in the image dataset, it can determine whether the image is over-enlarged or over-reduced, and the image dataset can then be automatically scaled to match the actual size of the image-visible marker or optical code. This scaling, in turn, matches the image dataset to the correct real-world size of the patient's body or actual human tissue.

[0049] In another configuration of the present technology, measurements of multiple image-visible markers manufactured to be the same size can be obtained from an image dataset. If slight errors or discrepancies in the size and length of the image-visible markers are expected, as often occurs in the real world (e.g., five markers each expected to be 10 mm long tubes), the average values ​​of the multiple image-visible markers can be averaged together to find the average size of the markers. As a result, a calculation can be made to determine whether the image dataset is too large or too small on average (e.g., off by 0.5 mm, 1 mm, or 3 mm in scaling based on the average of five markers), or if the image dataset does not match either the pre-known measurements or the measurements measured from the images captured by the camera. Thus, the scaling and comparison discussed above can be performed based on the average size of the image-visible markers.

[0050] This scaling aspect of the present technology can also be applied to the use of image-visible radiopaque materials used with other imaging modalities, not just MRI. For example, radiopaque material can be printed on an optical code. Examples of radiopaque materials may include, but are not limited to, metal spheres, liquid spheres, radiopaque plastic, metal-impregnated rubber, metal strips, paramagnetic materials, and portions of metallic ink. Radiopaque materials can be printed to a resolution of less than one-tenth of a millimeter (0.1 mm). The above-described scaling techniques can be used with radiopaque markers for many types of imaging modalities, such as X-ray, CT scan, or other imaging modalities capable of acquiring image datasets containing image-visible markers. A more detailed list of non-optical imaging modalities to which the magnification and image-visible markers may be applied includes computerized tomography (CT) scan modalities, X-ray modalities, positron emission tomography (PET) modalities, ultrasound modalities, fluorescence modalities, infrared thermography (IRT) modalities, 3D mammography, or single-photon emission computed tomography (SPECT) scan modalities. For example, image-visible markers used in other modalities may be radiopaque markers, sound-generating structures for ultrasound, contrast agents, or contrast agents in tubes, etc.

[0051] In one configuration of the present technology, when the patient's body is viewed through an AR headset, the system can identify an optical code (e.g., AprilTag) on ​​the patient's body as one or more points in space, and it is desirable to know the point of the optical code (e.g., AprilTag) down to submillimeter accuracy. However, because the optical code covers many points in space, it is not a single point in space. As a result, when viewing the optical code through an AR headset, there may be perspective issues, optical imperfections, parallax issues, and foreshortening that arise when viewing and identifying the location of the optical code and associated 3D points. As a more specific example, the distance from the optical sensor or camera on the AR headset from a first point on the optical code may be farther than a second point on the optical code, which may slightly shift the position estimate of the image dataset relative to the patient's body. To improve accuracy in registration to millimeters and submillimeters, the optical code can be treated as a single point in space rather than a point cloud, which is subject to visual distortion.

[0052] Aligning the image datasets can be performed by reducing the visible optical code to a single point in space and aligning that single point with an image visible marker that is also reduced to a second single point in the 3D space seen by the AR headset. Rather than using many points across the optical code for alignment, the optical code can be identified as a single point in 3D space. The coordinates of the edges (e.g., corners or other optical code features) can be identified, and mathematical lines (e.g., diagonals, bisectors, etc. of a square or rectangle) can be drawn between the edges of the optical code, and the optical code can then be reduced to a single point in space at the point where the lines intersect.

[0053] Reducing the optical code to a single point in space can help correct errors created by parallax and geometric distortion. Using multiple points of the optical code for alignment can also be more accurate than using a single point. In some situations, the measurement of one or more points of the optical code, such as a corner, may vary by a few millimeters depending on the level at which the edge detection threshold is set in the AR headset or the optical distortion present at various points in the visible image. This technique can help improve alignment and correct the potential problems discussed above.

[0054] 8 illustrates using an augmented reality (AR) headset to find and use the center of an optical code 854 to align or co-locate an image dataset 880 with a person's body 850. One or more optical codes 854 printed on a material 852 and associated with the person's body can be identified in a camera image captured by the AR headset's visible light camera. The optical codes on the person's 850 body can be positioned in a fixed location relative to an image visible marker 860 (e.g., a tube or radiopaque material).

[0055] One or more edges of the optical code 854 associated with the body of the person 850 in the visual image data can also be identified. A center point 856 of the optical code can be identified using one or more edges of the optical code 854.

[0056] One method for calculating the center, geometric center, or centroid of the optical code 854 can be to calculate a first line between a first point on the edge of the optical code and a second point on the edge of the optical code 854 to form a first diagonal 870 of the optical code. The first and second points can be corners of the optical code. A second line 872 can also be calculated between a third point on the edge and a fourth point on the edge (e.g., a corner) of the optical code to form a second diagonal of the optical code that intersects with the first line. A center point 856 of the optical code where the first and second lines intersect can then be determined. The image dataset can be aligned with the person's body using the center points of the optical code and the image visible code because the distance between the image visible code and the center of the optical code is fixed and known.

[0057] In one example, the AprilTag can have radiopaque ink printed on the back of the AprilTag forming an image-visible marker 860 that can be visible on a CT scan, X-ray, or imaging modality that can detect radiopaque markers. The AprilTag can be automatically found and uniquely identified because each AprilTag has a different pattern and the optical code has an image-visible marker 860 printed on the back of the optical code material or backing that is visible on a CT scan. In one example, the radiopaque material can be barium or bismuth. Additionally, tubes containing paramagnetic material can also be used as image-visible markers for MR images.

[0058] As shown in the figure, finding the center or centroid of an optical code can be calculated for an optical code that is square or rectangular, although the center or centroid of an optical code of any shape, such as a circle, triangle, star, hexagon, polygon, irregular shape, or any other shape, can be calculated.

[0059] Locating the center of gravity 866 of the optical code can correct for errors created by surface curvature on the surface of a human patient. Placing an X through the visible optical code helps to better identify a single location of the optical code to use in the alignment process despite any surface curvature of the optical code due to the optical code being attached to the surface of the patient's body. Locating the center of gravity using any known method can also help correct for any curvature or other distortion of the optical code attached to the patient. For example, locating the center of gravity, or other clearly defined single point, of the optical code can correct alignment issues, help identify the optical code, and improve accurate tracking of the optical code.

[0060] Multiple center points can also be used for registration. In this case, the center points of multiple optical codes in the visual image data can be found. An average center point of the center points of the multiple optical codes can be calculated. The image dataset can be registered with the person or patient using the average center point of the multiple optical codes.

[0061] Locating the corners of a box or rectangle containing an optical code can also be made more accurate by using a QR code, 2D barcode, or April tag (shown in FIG. 2) that uses dark or black boxes on the corners of the optical code. The boxes allow the corners of the optical code to be clearly identified using edge detection, and this corner identification can allow the center coordinates of the optical code to be more accurately determined. The use of boxes or dark visual highlights on the edges of the visual code makes the identification of the corners of the visual code more invariant to visual thresholds used to find the edges (e.g., edge detection) or varying lighting conditions.

[0062] Locating the center of an optical code can also help correct errors created by contrast resolution and threshold detection issues. An optical code or visual tag may be accurate in printed form, but when the optical code is imaged, it may be blurry or lack contrast. By using a diagonal or bisector with a square or rectangle, or by finding the center of a circle, the location of the optical code can then be more easily determined or encoded. If the optical code were circular, as discussed, the center of the circle could be found and a single point used for alignment.

[0063] Locating the center point of the optical code can also help more accurately capture optical codes that are not relatively large compared to the image data set. If MR imaging is captured with a 256x192 matrix (lower resolution compared to CT), the optical code is preferably not excessively large compared to the size of a person, because larger optical codes may have more geometric distortion on the patient's body. Therefore, more accurate center points can be found for small optical codes.

[0064] 9a is a flowchart of an exemplary method of using an augmented reality headset to colocate an image dataset and a patient's body using a tube containing an image-visible marking fluid. In these and other embodiments, the method can be performed by one or more processors based on one or more computer-readable instructions stored on one or more non-transitory computer-readable media.

[0065] 9a illustrates a method for coexisting a human body with an image dataset using an augmented reality (AR) headset. As in block 904, the method may include using an optical sensor of the AR headset to identify one or more optical codes associated with the human body. The optical code for the human body may have an image-visible marker that includes an image-visible contrast agent (i.e., a paramagnetic fluid) in a tube or sealed container that is in a fixed position relative to the optical code.

[0066] As at block 906, image visible markers with image visible contrast agent in the tube can be identified in an image dataset acquired using a medical imaging device (e.g., an MRI scanner). As at block 908, the image dataset can be aligned with the person's body when viewed through an AR headset using one or more optical codes on the person's body. Furthermore, the image dataset can be aligned with the image visible contrast agent in the tube relative to the optical codes using fixed positions of the image visible markers when referencing representations of the image visible markers with image visible contrast agent in the tube in the image dataset.

[0067] 9b illustrates a method for coexisting an image dataset with a person's body using an augmented reality (AR) headset. As at block 920, visual image data of a portion of the person's body can be acquired using an optical sensor or camera of the AR headset. As at block 922, one or more optical codes associated with the person's body can be identified within the visual image data. The optical code for the person's body can have image visible markers positioned at fixed positions relative to the optical code.

[0068] Known sizes of geometric attributes of image visible markers associated with the person's body may be identified, as in block 920. The image visible markers may appear in an image dataset when captured using a medical imaging device.

[0069] As at block 926, the measured sizes of the geometric attributes of the image visible markers in the image dataset may be compared to known sizes of the geometric attributes of the image visible markers to determine a calculated difference in size between the measured and known sizes of the geometric attributes of the image visible markers in the image dataset. As at block 928, the scaling of the image dataset may be corrected. The image dataset may be aligned with the person's body when viewed through the AR headset and the image visible markers using one or more optical codes on the person's body. This alignment may occur based in part on the calculated difference in size between the known and measured sizes of the geometric attributes of the image visible markers in the image dataset.

[0070] 9c illustrates a method for coexisting an image dataset with a human body using an augmented reality (AR) headset. As at block 932, the method may include identifying one or more optical codes associated with the human body using an optical sensor of the AR headset. The optical codes for the human body may be positioned at a fixed position relative to the image visible markers.

[0071] As at block 934, an edge of an optical code associated with a person's body in the visual image data may be identified or detected. Points on the edge may then be selected. As at block 936, a center point of the optical code may be identified using points on one or more edges of the optical code. As at block 940, the image dataset may be aligned with the person's body using a first center point of the optical code and a second center point of the image visible code.

[0072] 10 illustrates an exemplary system that may be used to use an augmented reality (AR) headset to cause an image dataset to coexist with the body of a person being viewed through the AR headset. The system 1000 may include a camera device 1002, an augmented reality system 1020, a display device 1030, and multiple databases or data stores. The system 1000 may also include one or more processors, memory, file systems, communication units, operating systems, and user interfaces, which may be communicatively coupled together. In some embodiments, the system 1000 may be, for example, a desktop computer, a server computer, a laptop computer, a smartphone, a tablet computer, an embedded computer, an AR headset, a VR headset, etc.

[0073] The camera device 1002 can be configured to capture visual data. In one example, the camera device 1002 can be used to capture visual data during a medical procedure or a medical examination. The visual data captured by the camera device 1002 can include images of a person's body (or body portion), optical codes, image visible markers, medical implements (e.g., medical instruments, implants, etc.). The camera device 1002 can transmit the captured optical data to the augmented reality system 1020. The system can also include surface sensors, optical sensors, infrared sensors, lidar sensors, or other sensors to detect and assist in the actual view or mapping of the actual view detected by the AR system. Any object or surface can be detected for an operating room, a patient, a room, a physical shape, a medical implement, or any other physical surroundings or object.

[0074] The augmented reality system 1020 may include an image processing engine 1022, a fiducial and alignment module 1024, an image generation module 1026, and an enhanced display buffer 1028. For example, the image processing engine 1022 receives captured visible image data from the camera device 1002 and analyzes the visible image data to identify one or more optical codes, objects, or people within the visible image data. A number of different techniques may be used to identify objects or people within the visible image data, including, but not limited to, feature extraction, segmentation, and / or object detection.

[0075] The image processing engine 1022 also identifies optical codes that may be affixed to both of the patient's bodies in the image. Once the image processing engine 1022 identifies an optical code (e.g., AprilTag, 2D barcode, QR code, and others), the image processing unit 1022 accesses the optical code database 1046 to retrieve information associated with the optical code. In some examples, the optical code is associated with an image-visible marker of a measured size, such as a tube containing a measured length of paramagnetic fluid. Furthermore, the optical code can be associated with a particular patient, a particular procedure, or a particular object.

[0076] In some embodiments, the fiducials and registration module 1024 cooperates with the image processing engine 1022 to reference the person's body and image datasets relative to one another, as described above. Additionally, the fiducials and registration module 1024 can use optical code information in the medical object database 1044 to properly identify the size and shape, and identification, of the optical code. Once the position and orientation of the patient's body is determined, the fiducials and registration controller 1026 can register any associated radiology images in the radiology image data 1042 with the patient's body. In some examples, the radiology images are received from the radiology image database 1042 based on the patient record in the patient record database 1040.

[0077] The image generation module 1026 can generate and display within the display device 1030 graphical data, virtual tools, 3D surgical paths, 3D coloring or shading of masses or organs, or highlighting of masses, organs, or targets as they are overlaid on top of the patient's body. In some examples, this information can be loaded into an expansion display buffer 1028. This information may then be transmitted to the display device 1030 for display to the user.

[0078] In one example, the patient database 1040 includes multiple patient records. Each patient record may include one or more medical procedures performed on the patient. The patient records may also include notes, instructions, or plans for the medical procedures. The patient records may also be associated with one or more radiology images in the radiology image database 1042. In some examples, the radiology images include representations of image visible markers that allow the fiducials and registration module 1026 to properly register the image dataset with the patient's body using fixed positions of the optical code relative to the image visible markers. In one example, the medical object data 1044 includes information describing medical items, including medical instruments, implants, and other objects.

[0079] In some embodiments, the augmented reality system may be located on a server, which may be any computer system capable of functioning in conjunction with an AR headset or display device 1030. In such embodiments, the server may be configured to communicate with the AR headset over a computer network to communicate image data to the AR headset or receive data from the AR headset.

[0080] 11 illustrates a computing device 1110 on which modules of the present technology may be executed. A computing device 1110 is shown on which a high-level example of the present technology may be executed. The computing device 1110 may include one or more processors 1112 in communication with a memory device 1112. The computing device may include a local communication interface 1118 for components within the computing device. For example, the local communication interface may be a local data bus and / or any associated address or control bus as desired.

[0081] The memory device 1112 may include modules 1124 executable by the processor 1112 and data for the modules 1124. The modules 1124 may perform the functions described above. A data store 1122 may also be located within the memory device 1112 for storing data related to the modules 1124 and other applications, along with an operating system executable by the processor 1112.

[0082] Other applications may also be stored in the memory device 1112 and executable by the processor 1112. The components or modules discussed herein may be implemented in software using a high-level programming language that is compiled, interpreted, or executed using a hybrid approach.

[0083] A computing device may also have access to I / O (input / output) devices 1114 that are usable by the computing device. One example of an I / O device is an available display screen for displaying output from the computing device. Other known I / O devices may be used with the computing device as desired. Networking devices 1116 and similar communication devices may also be included in the computing device. The networking devices 1116 may be wired or wireless networking devices that connect to the Internet, a LAN, a WAN, or other computing networks.

[0084] Components or modules shown as stored in memory device 1112 may be executed by processor 1112. The term "executable" may refer to a program file in a format that can be executed by processor 1112. For example, a program in a high-level language may be compiled into machine code in a format that can be loaded into a random access portion of memory device 1112 and executed by processor 1112, or source code may be loaded by another executable program and interpreted to generate instructions in a random access portion of memory to be executed by the processor. An executable program may be stored in any portion or component of memory device 1112. For example, memory device 1112 may be a random access memory (RAM), a read-only memory (READ-ONLY MEMORY), or a programmable logic (PLC). The memory may be a read-only memory (ROM), flash memory, a solid-state drive, a memory card, a hard drive, an optical disk, a floppy disk, a magnetic tape, or any other memory component.

[0085] The processor 1112 may represent multiple processors, and the memory 1112 may represent multiple memory units operating in parallel with the processing circuitry. This may provide parallel processing channels for processing and data within the system. The local interface 1118 may be used as a network to facilitate communication between any of the multiple processors and the multiple memories. The local interface 1118 may use additional systems designed to coordinate communications, such as load balancing, bulk data transfer, and similar systems.

[0086] Some of the functional units described herein have been labeled as modules to more particularly emphasize their implementation independence. For example, a module may be implemented as a hardware circuit comprising custom VLSI circuits or gate arrays, off-the-shelf semiconductors such as logic chips, transistors, or other discrete components. A module may also be implemented in programmable hardware devices such as field programmable gate arrays, programmable array logic, programmable logic devices, etc.

[0087] Modules may also be implemented in software for execution by various types of processors. An identified module of executable code may comprise one or more blocks of computer instructions, which may be organized as, for example, an object, a procedure, or a function. Nevertheless, the executable files of an identified module need not be physically located together, but may comprise heterogeneous instructions stored in different locations that make up the module and that, when logically combined together, achieve the purpose stated for the module.

[0088] Indeed, a module of executable code may be a single instruction, or many instructions, and may even be distributed across several different code segments, among different programs and across several memory devices. Similarly, operational data may be identified and illustrated herein within modules and may be embodied in any suitable form and organized within any suitable type of data structure. Operational data may be collected as a single data set or distributed across different locations, including across different storage devices. Modules may be passive or active, including agents operable to perform a desired function.

[0089] The techniques described herein may be stored on computer-readable storage media, including volatile and nonvolatile, removable and non-removable media implemented in any technology for storage of information such as computer-readable instructions, data structures, program modules, or other data, including, but not limited to, RAM, ROM, EEPROM, flash memory or other memory technology, CD-ROM, digital versatile disk (DVD) or other optical storage devices, magnetic cassettes, magnetic tape, magnetic disk storage devices or other magnetic storage devices, or any other computer storage medium that can be used to store the desired information and the techniques described.

[0090] Devices described herein may also include communications or networking equipment and connections that allow the devices to communicate with other devices. A communications connection is an example of a communications medium. Communications media typically embodies computer-readable instructions, data structures, program modules, and other data in a modulated data signal, such as a carrier wave or other transport mechanism, and includes any information delivery media. A "modulated data signal" means a signal that has one or more of its characteristics set or changed in such a manner as to encode information in the signal. By way of example, and not limitation, communications media include wired media, such as a wired network or direct-wired connection, and wireless media, such as acoustic, radio frequency, infrared, and other wireless media. As used herein, the term computer-readable media includes communications media.

[0091] Reference will be made to the examples illustrated in the drawings, and specific language will be used herein to describe the same. It will nevertheless be understood that no limitation of the scope of the technology is thereby intended. Alterations and further modifications of the features illustrated herein, and additional applications of the embodiments illustrated herein that will occur to those skilled in the art in possession of this disclosure, should be considered within the scope of the present specification.

[0092] Furthermore, the described features, structures, or characteristics may be combined in any suitable manner in one or more embodiments. In the preceding description, numerous specific details, such as embodiments of various configurations, have been provided to provide a thorough understanding of embodiments of the described technology. However, those skilled in the art will recognize that the technology can be practiced without one or more of the specific details, or with other methods, components, devices, etc. In other instances, well-known structures or operations have not been shown or described in detail to avoid obscuring aspects of the technology.

[0093] Although the present subject matter has been described in language specific to structural features and / or operations, it is to be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features and operations described above. Rather, the specific features and operations described above are disclosed as exemplary forms of implementing the claims. Numerous modifications and alternative arrangements may be devised without departing from the spirit and scope of the described technology. The technical ideas included in the present disclosure are described below. (Appendix 1) 1. A method for coexisting an image dataset with a human body using an augmented reality (AR) headset, comprising: using an optical sensor of the AR headset to identify one or more optical codes associated with the body of the person, the optical code for the person's body having an image visible marker including an image visible contrast agent in a tube or sealed container in a fixed position relative to the optical code; identifying the image visible marker comprising the image visible contrast agent within the tube in the image dataset acquired using a medical imaging device; and aligning the image dataset with the body of the person using one or more optical codes on the body of the person when viewed through the AR headset and using the fixed positions of the image visible markers having the image visible contrast agent in the tube relative to the optical codes when referencing representations in the image dataset of the image visible markers having the image visible contrast agent in the tube. (Appendix 2) 2. The method of claim 1, further comprising: the imaging-visible contrast agent being an MRI (magnetic resonance imaging)-visible contrast agent. (Appendix 3) 3. The method of claim 2, wherein the imaging contrast agent is a paramagnetic fluid. (Appendix 4) 3. The method of claim 2, wherein the image-visible contrast agent is encapsulated within the tube of a defined length or a tube of variable length. (Appendix 5) 5. The method of claim 4, wherein the defined length of the tube identifies an optical code with a defined 2D data encoding that is visible using the optical sensor of the AR headset. (Appendix 6) 6. The method of claim 5, further comprising associating the optical code visible using the optical sensor of the AR headset with the defined length of the tube by linking the defined length of the tube with the optical code in a data store. (Appendix 7) 2. The method of claim 1, wherein the tube is a formed tube in the shape of at least one of lines, a crossing structure, concentric circles, or a lattice structure. (Appendix 8) 4. The method of claim 3, wherein the tube has a two-dimensional (2D) cross-sectional shape that is at least one of circular, triangular, square, rectangular, elliptical, hexagonal, octagonal, oval, or polygonal. (Appendix 9) 4. The method of claim 3, wherein the tube is a plurality of tube segments, each having a distinct length. (Appendix 10) 2. The method of claim 1, wherein the image-visible contrast agent is gadolinium solution, copper sulfate, iron solution, manganese, vitamin E oil, or a paramagnetic material. (Appendix 11) 2. The method of claim 1, further comprising automatically retrieving the image dataset for the body of the person using one or more optical codes on the body of the person. (Appendix 12) 2. The method of claim 1, wherein the tube is at least one of a plastic tube, a glass tube, or a non-metallic tube. (Appendix 13) 1. A method for producing an encapsulated contrast agent and an optical code, comprising: sealing a first end of a plastic tube; cutting a length of said plastic tubing for use in MRI (Magnetic Resonance Imaging) data acquisition; filling the plastic tube with an image-visible contrast agent that can be captured in an MRI image dataset; sealing a second end of the plastic tube containing the imaging contrast agent; and attaching one or more optical cords to the plastic tube using a joining structure to maintain a fixed position between the plastic tube and the one or more optical cords. (Appendix 14) 14. The method of claim 13, wherein the plastic tube has a cross-sectional shape defined by a circle, an oval, a square, a rectangle, a semicircle, a wave, or a polygon. (Appendix 15) 14. The method of claim 13, further comprising filling the plastic tube with the imaging-visible contrast agent, which is an MRI (magnetic resonance imaging)-visible contrast agent. (Appendix 16) 14. The method of claim 13, further comprising filling the tube with an imaging contrast agent that is a paramagnetic fluid. (Appendix 17) 14. The method of claim 13, further comprising filling the tube with the image-visible contrast agent, which is gadolinium solution, copper sulfate, iron solution, manganese, or vitamin E oil. (Appendix 18) 1. A method for coexisting an image dataset with a human body using an augmented reality (AR) headset, comprising: detecting visual image data of a portion of the body of the person using an optical sensor of the AR headset; identifying one or more optical codes associated with the body of the person, the optical codes for the body of the person having image visible markers positioned in fixed positions relative to the optical codes; identifying a known size of a geometric attribute of the image visible marker associated with the body of the person, the image visible marker appearing in the image dataset; comparing the measured size of the geometric attribute of the image visible marker in the image dataset with the known size of the geometric attribute of the image visible marker to determine a calculated difference in size between the measured size and the known size of the geometric attribute of the image visible marker in the image dataset; and correcting a magnification of the image dataset to be aligned with the body of the person when viewed through the AR headset and the image visible markers using one or more optical codes on the body of the person based at least in part on the calculated difference in size between the known size and the measured size of the geometric attributes of the image visible markers. (Appendix 19) 19. The method of claim 18, further comprising increasing or decreasing the magnification of the image dataset to match the measured size of the geometric attribute of the image visible marker in the image dataset to the known size of the geometric attribute of the image visible marker. (Appendix 20) 19. The method of claim 18, further comprising increasing the magnification of the image dataset when viewed through the AR headset, wherein the known size of the optical code is greater than the measured size of the image visible markers in the image dataset. (Appendix 21) 19. The method of claim 18, further comprising reducing the magnification of the image dataset when viewed through the AR headset, wherein the known size of the optical code is smaller than the measured size of the image visible markers in the image dataset. (Appendix 22) 19. The method of claim 18, wherein the sizes of the geometric attributes are pre-measured values ​​stored in a data store accessible to the AR headset. (Appendix 23) 19. The method of claim 18, wherein the size of the geometric attribute is measured from image visible markers visible on the person's body acquired by the AR headset. (Appendix 24) 19. The method of claim 18, wherein the geometric attribute of the optical code is at least one of a length of a tube of paramagnetic markers, a distance between two metal markers of the optical code, a radius encompassing multiple metal markers of the optical code, a diameter of the image visible marker, or a geometric measurement of the image visible marker. (Appendix 25) 1. A method for coexisting an image dataset with a human body using an augmented reality (AR) headset, comprising: identifying, using an optical sensor of the AR headset, one or more optical codes associated with the body of the person, wherein the optical codes for the person's body are positioned in a fixed position relative to an image visible marker; identifying one or more edges of an optical code associated with the body of the person; identifying a center point of the optical code using the one or more edges of the optical code; and using the center point of the optical code and the image visible code to register the image dataset with the body of the person. (Appendix 26) 26. The method of claim 25, wherein the center point is a calculated center of gravity of the optical code. (Appendix 27) calculating a first line between a first point on the edge of the optical code and a second point on the edge of the optical code to form a first diagonal of the optical code; calculating a second line between a third point on the edge of the optical code and a fourth point on the edge, the second line forming a second diagonal of the optical code and intersecting the first line; 26. The method of claim 25, further comprising identifying a center point of the optical code where the first line and second line intersect. (Appendix 28) 26. The method of claim 25, further comprising determining a location for displaying the image dataset via the AR headset by using the center point of the optical code relative to the image visible marker and a fixed distance between the center point of the optical code and the image visible marker. (Appendix 29) locating the center points of a plurality of optical codes within the visual image data; calculating an average center point of the center points of the plurality of optical codes; 26. The method of claim 25, further comprising: using the mean center point of the plurality of optical codes to align the image dataset with a human body.

Claims

1. 1. A method for coexisting an image dataset with a human body using an augmented reality (AR) headset, comprising: detecting visual image data of a portion of the body of the person using an optical sensor of the AR headset; identifying one or more optical codes associated with the body of the person, the optical codes for the body of the person having image visible markers positioned in fixed positions relative to the optical codes; identifying a known size of a geometric attribute of the image visible marker associated with the body of the person, the image visible marker appearing in the image dataset; comparing the measured size of the geometric attribute of the image visible marker in the image dataset with the known size of the geometric attribute of the image visible marker to determine a calculated difference in size between the measured size and the known size of the geometric attribute of the image visible marker in the image dataset; and correcting a magnification of the image dataset to be aligned with the body of the person when viewed through the AR headset and the image visible markers using one or more optical codes on the body of the person based at least in part on the calculated difference in size between the known size and the measured size of the geometric attributes of the image visible markers.

2. 2. The method of claim 1, further comprising increasing or decreasing the magnification of the image dataset to match the measured size of the geometric attribute of the image visible marker in the image dataset to the known size of the geometric attribute of the image visible marker.

3. 10. The method of claim 1, further comprising increasing the magnification of the image dataset when viewed through the AR headset, wherein the known size of the optical code is larger than the measured size of the image visible markers in the image dataset.

4. 10. The method of claim 1, further comprising reducing the magnification of the image dataset when viewed through the AR headset, wherein the known size of the optical code is smaller than the measured size of the image visible markers in the image dataset.

5. The method of claim 1 , wherein the sizes of the geometric attributes are pre-measured values ​​stored in a data store accessible to the AR headset.

6. The method of claim 1 , wherein the size of the geometric attribute is measured from image visible markers visible on the person's body acquired by the AR headset.

7. 2. The method of claim 1, wherein the geometric attribute of the optical code is at least one of a length of a tube of paramagnetic markers, a distance between two metal markers in the optical code, a radius encompassing multiple metal markers in the optical code, a diameter of the image visible marker, or a geometric measurement of the image visible marker.

8. 1. A method for coexisting an image dataset with a human body using an augmented reality (AR) headset, comprising: using an optical sensor of the AR headset to identify one or more optical codes associated with the body of the person, wherein the optical codes for the person's body are positioned in a fixed position relative to an image visible marker; identifying one or more edges of an optical code associated with the body of the person; identifying a center point of the optical code using the one or more edges of the optical code; and using the center point of the optical code and the image visible code to register the image dataset with the body of the person.

9. The method of claim 8 , wherein the center point is a calculated center of gravity of the optical code.

10. calculating a first line between a first point on the edge of the optical code and a second point on the edge of the optical code to form a first diagonal of the optical code; calculating a second line between a third point on the edge of the optical code and a fourth point on the edge, the second line forming a second diagonal of the optical code and intersecting the first line; The method of claim 8 , further comprising: identifying a center point of the optical code where the first line and the second line intersect.

11. 9. The method of claim 8, further comprising determining a location for displaying the image dataset through the AR headset by using the center point of the optical code relative to the image visible marker and a fixed distance between the center point of the optical code and the image visible marker.

12. locating the center points of a plurality of optical codes within the visual image data; calculating an average center point of the center points of the plurality of optical codes; The method of claim 8 , further comprising: registering the image dataset with a human body using the mean center point of the plurality of optical codes.

Images