Medical Image Registration in Augmented Reality Displays

Abstract

An AR headset for coexisting an image dataset with a human body is described. An image dataset of a portion of the body may be identified. The image dataset may include a virtual anatomical marker for identifying an anatomical location. The spatial location may be identified using a physical pointer object in the field of view of the AR headset. The spatial location of the visible anatomical feature identified using the physical pointer object may be recorded. The position registration may be triggered based on an input that the physical pointer object is located at a visible anatomical feature of the human body. Furthermore, the image dataset may be aligned with the human body using the recorded spatial location of the visible anatomical feature relative to the virtual anatomical marker in the image dataset.

Description

[Background technology]

[0001] Mixed reality or augmented reality is an area of ​​computing technology that can combine images or views from the physical world and virtual computing worlds into a mixed reality world. In mixed reality, people, places, and objects from the physical and virtual worlds become a blended environment. Mixed reality experiences can be delivered through the use of virtual reality (VR) or augmented reality (AR) headsets, as well as through existing commercial or custom software.

[0002] Augmented reality (AR) is an example of mixed reality in which a live, direct or indirect view of a physical, real-world environment is augmented or supplemented with computer-generated sensory input, such as images, audio, video, graphics, or other data. Such augmentation can occur when a real-world location is viewed, and augmented graphics can be realized when associated with visible environmental elements. With the help of advanced AR techniques (e.g., adding computer vision and object recognition), information about the real world around the user can become interactive and / or digitally modified (e.g., via computer graphic overlays).

[0003] A problem faced in using an AR system or AR headset is identifying the position and orientation of a physical object with high accuracy. Similarly, aligning the position of a virtual element with a physical object in a live view of a real-world environment can be difficult. For example, aligning a virtual object with a physical object when viewed through an AR headset can be difficult because the luminescence or brightness of the virtual object obscures the underlying real-world object with which the virtual object is to be aligned. While aligning a virtual object with a physical object to within approximately a few centimeters may be useful for entertainment and less demanding applications, higher positioning and alignment resolution for AR systems may be desirable in scientific, engineering, and medical fields. As a result, the positioning and alignment process for AR used in scientific, engineering, and medical fields may be performed manually, which can be time-consuming, cumbersome, and inaccurate. [Brief explanation of the drawings]

[0004] [Figure 1A] 1 illustrates an exemplary augmented reality (AR) environment in which an image dataset of a patient may be aligned to a real view of the patient using a physical pointer object that identifies spatial locations that align with virtual anatomical markers in the image dataset. [Figure 2] 1 illustrates an example of a physical pointer object with a 2D barcode. [Figure 3A] 1 illustrates an exemplary top view of an image dataset of a patient or person that can be aligned with an actual view of the patient using a spatial location identified using a physical pointer object. [Figure 3B] 1 illustrates an exemplary top view of multiple optical codes used on a physical pointer object. [Figure 3C] 10 illustrates an embodiment of rotating or orbiting a physical pointer object around a point to be located. [Figure 3D] 1 illustrates an example of a virtual plane that can be formed using a physical pointer object and a camera on an AR headset. [Figure 3E] 1 illustrates an example of creating a virtual plane using a physical pointer object. [Figure 3F] 1 illustrates an example in which a physical pointing device is rotated around a point to generate multiple planes that are used to register the spatial point. [Figure 3G] 1 illustrates an example of a physical pointing device for pointing to a physical anatomical point beneath the skin of a patient's body. [Figure 3H] 1 illustrates fiducials or anatomical points marked on two-dimensional and three-dimensional image datasets. [Figure 3I] 1 illustrates fiducials or anatomical points marked on two-dimensional and three-dimensional image datasets. [Figure 4A] 1 illustrates an example of a cross-sectional view of a location on a vertebra of a human body where a virtual anatomical marker has been created by a medical professional. [Figure 4B] 1 illustrates an exemplary cross-sectional side view of locations in the spinal region of a person with virtual anatomical markers. [Figure 4C] 1 illustrates an exemplary posterior view of locations in the spinal region of a person with virtual anatomical markers. [Figure 5] 1 illustrates an exemplary use of a physical pointing object to enable alignment of a virtual anatomical marker created by a medical professional in an image dataset to a spatial location identified by the physical pointer object. [Figure 6] Using an augmented reality (AR) headset, we demonstrate a method for using a physical pointer object to co-locate an image dataset with a person's body. [Figure 7] We illustrate a method for using an augmented reality (AR) headset to co-locate an image dataset with a person's body using a physical pointer wand. [Figure 8]We illustrate a method for using an augmented reality (AR) headset to co-locate an image dataset with a person's body using a finger as a physical pointer object. [Figure 9] 1 illustrates an exemplary system that may be used to register an image dataset to a human body. [Figure 10] FIG. 1 is a block diagram illustrating an example of a computing system for processing the present technology. DETAILED DESCRIPTION OF THE INVENTION

[0005] A technique is provided for coexisting an image dataset with a human body using an augmented reality (AR) headset. The technique may use a physical pointer object to identify a spatial location for alignment with a virtual anatomical marker in the image dataset. The virtual anatomical marker may be created or marked in the image dataset by a medical professional. The physical pointer object may have an optical code thereon, which may be used to determine the location of the physical pointer object, and then the location of a pointer or point (e.g., tip) of the physical pointer object in three-dimensional (3D) space viewed by the AR headset. The AR headset may identify one or more optical codes (e.g., AprilTag, QR code, or 2D optical barcode) within the field of view of the AR headset's camera during a medical procedure. The physical pointer object may be an object such as a wand, trocar, needle, or finger of the medical professional or user.

[0006] The image dataset may be registered with a person's body during a medical procedure by aligning virtual anatomical markers with spatial locations identified using a physical pointer object that are touched or marked by a tip or point on the physical pointer object. The image dataset may be a previously acquired image of a portion of the person's body using a non-optical imaging modality. Examples of non-optical imaging modalities may be an MRI (magnetic resonance imaging) modality, a CT (computed tomography) scanning modality, an X-ray modality, a Positron Emission Tomography (PET) modality, an ultrasound modality, a fluorescence modality, an Infrared Thermography (IRT) modality, a 3D mammography, or a Single-Photon Emission Computed Tomography (SPECT) scanning modality, etc. The image dataset may be registered to the person's body using virtual anatomical markers (e.g., image visible markers) that are registered to a spatial location defined by the tip of a physical pointer object or a spatial location defined at a known distance from the tip of a physical pointer object.

[0007] The term "aligned" can mean that the virtual anatomical marker and the spatial point are aligned within a few millimeters, or that the alignment of the virtual anatomical marker can be within a set distance (e.g., 1 cm below the surface) from the spatial point. For example, a virtual anatomical marker can be aligned with the tip of a physical pointer object, but the virtual anatomical marker can be aligned to be a few millimeters further below the skin or bone, etc. Virtual anatomical markers are markers, tags, or notations that are added to and found in datasets captured using non-visible imaging modalities, such as captured radiology images or image datasets that may not be visually visible to an AR headset.

[0008] 1 illustrates an example of an augmented reality (AR) environment 100 in which an image dataset 116 of a patient or person 106 may be aligned with an actual view of the patient 106 using a spatial location identified using a physical pointer object 118 in the field of view of an optical sensor of an AR headset. For example, the spatial location may be a spot, area, or shape in space identified using the physical pointer object. In another example, a target location or target area may be considered a spatial location.

[0009] An image dataset of a portion of a person's body may be acquired using a medical imaging device, and the image dataset may have virtual anatomical markers for identifying anatomical features. In one embodiment, the image-visible markers may be created or placed by a medical professional who marks the image using a virtual marking tool and virtual graphics to create the virtual anatomical markers using computer graphics. For example, locations for the virtual anatomical markers in the image dataset may be defined before a medical procedure using a computing device or workstation controlled by the medical professional. The virtual anatomical markers in the image dataset may be virtual graphical marks that are dots, circles, lines, curves, splines, crosshairs, points, planes, callouts, two-dimensional (2D) graphic objects, or three-dimensional (3D) graphic objects. The medical professional or other individual may trace a curve or shape used for registration, such as the shape of a patient's anatomical feature. As used herein, the term anatomical feature may refer to any anatomical aspect of a person or patient and does not necessarily refer to a unique anatomical feature.

[0010] Alternatively, the image dataset may be populated with representations of virtual anatomical markers in machine-captured images that capture structures of the human body using non-optical imaging modalities. For example, image-visible markers such as image-visible codes, image-visible fluid-filled structures, image-visible fluids in containers or capsules (e.g., MRI beads or spheres), radio-density markers (e.g., CT radio-opaque markers), radioisotopes, or image-visible objects (e.g., image-visible materials) may be used to create virtual anatomical markers in the image dataset. For other imaging modalities such as those mentioned above, virtual anatomical markers may be created by providing an appropriate contrast agent in a container that is visible in images captured by the respective modality (e.g., MRI, CT, nuclear-based image acquisition, etc.).

[0011] The physical pointer object 118 may be a metal wand, a plastic wand, a trocar, a needle, a physical object with a pointing tip, or another medical pointer having a detectable pointer location in three-dimensional (3D) space. The physical pointer object 118 may have an optical code 199 affixed to the physical pointer (e.g., a wand or a finger) to identify the spatial location of the physical pointer object 118.

[0012] Alternatively, the optical code 199 may be permanently printed on or formed into the handle or other portion of the physical pointer object. For example, the optical code may be formed into the handle of the physical pointer object during an injection molding or 3D printing process to form recessed optical code regions representing light portions of the optical code and non-recessed optical code regions representing dark portions of the optical code (or vice versa). Additionally, the optical code may be embossed onto the handle of the physical pointer object using colored plastic, paint, or similar materials. The optical code 199 on the physical pointer object may also be identified in camera images captured by the AR headset 108 using a visible AprilTag or 2D barcode as the optical code. The spatial location of the physical wand may be recognized by the AR headset using the optical code. In an alternative configuration, the spatial location of the tip of the physical pointer object may be recognized by first detecting the edges of the physical pointer object. Once the edges of the physical pointer object are known, the tip of the physical pointer object or the pointer may be identified.

[0013] The optical code may also include data and / or values ​​embedded in the optical code. In one embodiment, the embedded data may be a value defining the medical instrument type, medical instrument model, medical instrument version, or medical instrument manufacturer. Additionally, each physical pointer object may have a unique code for the device. When the unique code for a physical pointer object is recognized, the AR headset may unlock the ability to use the physical pointer object or unlock pre-recorded navigation or guidance software for that physical pointer object. The data in the optical code may also be used to check that the correct medical instrument, image dataset file, and medical instrument shape are being used in a medical procedure. The value in the optical code of a physical pointer object may be used to verify licensing or payment for virtual medical procedure software or registration software, and / or the correct code may unlock a navigation software system. The value in the optical code of a physical pointer object may also be used in combination with optical codes in other medical instruments (e.g., an instrument tray, etc.) to verify licensing or payment for virtual medical procedure software or registration software, and / or the correct code may unlock a navigation software system. The data and values ​​embedded in the optical code may be encrypted code values ​​or data to protect keys or other valuable data in the stored data and values.

[0014] In one embodiment, the optical code 199 may be an AprilTag, a linear barcode, a matrix two-dimensional (2D) barcode, a Quick Response (QR) code, a Fourier Tag (e.g., any optical code including a Fourier code), or some combination thereof. AprilTags are two-dimensional barcodes that may be a visual reference system useful for augmented reality and / or camera calibration. AprilTags may be used to calculate the 3D position, orientation, and identity of the tag relative to a camera, sensor, or AR headset.

[0015] During use, the spatial location of a visible anatomical feature may be recorded as identified using the tip, pointer, or other point of the physical pointer object. The spatial location may be recorded upon receiving input to the AR headset 108 or another computing device, indicating that the physical pointer object is positioned at or pointing to a visible anatomical feature on a person's body. Furthermore, the spatial location may be defined as at least one of a point, a curve, a tracked two-dimensional (2D) area, or a tracked three-dimensional (3D) shape. Thus, the spatial location of the physical pointer object may further define a three-dimensional (3D) spatial location in X, Y, and Z coordinates (or polar coordinates) as recorded using the optical sensor of the AR headset. A medical professional may provide an indicator when a curve, area, or shape begins to be tracked and indicate when tracking of the curve, area, or shape is complete. By knowing the location of the tag on the physical pointer object, the location of the tip (e.g., needle tip or pointer tip) may be calculated at any time. The image dataset can then be aligned with the person's body using the recorded spatial locations of the visible anatomical features relative to the virtual anatomical markers in the image dataset. As a result, the image dataset can be displayed as visually overlaid on the person's body using an AR headset.

[0016] The environment 100 may include a physical space 102 (e.g., an operating room, a laboratory, etc.), a user 104 (e.g., a medical professional), a patient 106, a physical pointer object 118 having an optical code 199, and an AR headset 108 in communication with a server 112 via a computer network 110. A virtual user interface 114 and a virtual cursor 122 are shown in dashed lines to indicate that these virtual elements are generated by the AR headset 108 and are visible by the user 104 through the AR headset 108.

[0017] The image dataset 116 may be captured using a non-optical imaging modality. One example of a non-optical imaging modality used to capture the image dataset may be an MRI modality. In another example, the non-optical image or image dataset may be an image or image dataset including a combination of two or more forms of non-optical imaging modalities (e.g., two or more images combined together, combined segments of two or more non-optical images, a CT image fused with an MRI image, etc.). Each image dataset in a separate modality may have virtual anatomical markers in the separate image dataset, which may allow the PET image, CT image, MRI image, fluoroscopic image, etc. to be registered and / or referenced with a spatial point identified by a physical pointing object on the person's body in the AR system view.

[0018] The AR headset 108 may be an AR computing system that can augment a real-world view of the patient 106 (covered by hospital gown or fabric 107) with image datasets 116. For example, the AR headset 108 may be used by the user 104 to augment a real-world view of the patient 106 with a view of one or more 3D image datasets of the patient 106, including, but not limited to, bones (as illustrated in FIG. 1 ), muscles, organs, pathologies such as tumors, lymph nodes, or fluid 106b, or radiology images.

[0019] The AR headset 108 may enable the image dataset 116 (or a projection of the image dataset) to be dynamically reconstructed. Thus, as the user 104 moves around the patient 106, sensors in the AR headset 108 determine the user 104's location relative to the patient 106, and the patient's internal anatomical structures displayed using the image dataset can be dynamically reconstructed as the user 104 selects different orientations relative to the patient. For example, the user 104 may walk around the patient 106. The AR headset 108 can then augment the actual view of the patient 106 with one or more acquired radiology images or image datasets (MRI, CT scan, etc.) of the patient 106, so that both the patient 106a and the image dataset 116 of that patient 106a can be viewed by the user 104 from any angle (e.g., projected images or slices from the image dataset can also be displayed). The AR headset 108 may be a modified version of a Microsoft HOLOLENS, a Meta Company META 2, an Epson MOVERIO, a Garmin VARIA VISION, a Magic Leap MAGIC LEAP 1, or other AR headset.

[0020] The virtual user interface 114 generated by the AR headset 108 may include options for changing the display of the projected internal anatomical structures of the patient 106 from the image dataset 116 of the patient 106. The virtual user interface 114 may include other information that may be useful to the user 104. For example, the virtual user interface 114 may include information about the patient or medical implement 118 (e.g., medical instruments, implants, etc.) identified with an optical code. In another example, the virtual user interface 114 may include medical records or other medical data about the patient 106. In some configurations, the person's image data 116 or captured radiological data may be displayed by the AR headset 108 using the large image dataset 116 to display the patient's 106a's radiologically captured anatomical structures (e.g., bones 106b, tissue, blood vessels, fluid, etc.) from the image data. This image data may include axial, coronal, sagittal, or oblique slices of the image data. A slice may be a two-dimensional (2D) slice having a depth as well as a height and width (e.g., one or more layers of voxels), a three-dimensional (3D) slice, and / or a four-dimensional (4D) slice (a 3D image having a time series of images). The user 104 can control the virtual user interface 114 using hand gestures, voice commands, eye movements, a remote control (e.g., a finger clicker), a 3D mouse, a VR wand, a finger sensor, haptic technology, or other control methods.

[0021] The distal end point or tip on the elongated shaft of the physical pointer object distal to the handle may be used to identify the spatial location, although any other predefined point may be used on the physical pointer object to identify the spatial location. For example, if the physical pointer object is a triangle or another polygon, one of the points on the physical pointer object may be used as the tip to identify the spatial location. Similarly, the physical pointer object may be a cube or other shape, and there may be multiple pointers of various sizes or colors extending from the cube or other shape that may be used to identify the spatial location. For example, the size of the pointer or the color of the pointer may correspond to the spatial point to which it is set.

[0022] The image dataset may be automatically retrieved for the person's body using one or more optical codes on the person's body. Alternatively, the image dataset may be retrieved using optical codes on a physical pointer object, as images needed for a medical procedure may be associated with the physical pointer object.

[0023] In one example, a patient 106 may be moved, for example, from a medical imaging suite within a hospital to an operating room within the hospital. A user 104 (such as the medical professional in FIG. 1 ) may then use an AR headset 108 to determine the location of an optical code 199 on a physical pointing object 118. The AR headset 108 may then automatically retrieve image data of the patient 106 based on the optical code on the physical pointing object 118.

[0024] After detecting the optical code 199 in the 3D space 102, the AR headset 108 may automatically calculate the position of the physical pointing object 118. This automatic calculation may be based on the sensed position of the optical code 199 in the 3D space 102 and / or the known fixed position of the physical pointing object relative to the position of the optical code 199.

[0025] In a further configuration, a virtual or graphical pointer may extend from and be controlled by a physical pointer object, and the pointer (e.g., tip) of the virtual pointer may be used to define a spatial location. The physical pointer object may be a medical professional's finger identified by an AR headset. For example, if the medical professional's finger is detected as a physical pointer object by the AR headset, a virtual line or virtual object in 2D or 3D may be extended away from the medical professional's finger. The distal end of the virtual line or virtual object may be used as a pointer to define the location of a point, area, or shape identified as a spatial location. Similarly, a virtual pointer may extend from a wand, pointer, trocar, mobile device, joystick, or another physical pointer rod or pointer structure. As in this example, a virtual structure extended from the finger by identifying the finger may create a virtual light beam from the finger that may be used as a virtual laser pointer. This allows the medical professional to see the anatomical structures used at the spatial point, and the medical professional can virtually touch (e.g., aim at or virtually overlay) the selected point and register the point as a spatial location.

[0026] In an alternative configuration, the AR headset may have a virtual laser pointer without a physical pointer object, where the cursor is controlled by the positioning of the user's head within the AR headset. This virtual laser pointer may allow the user to point with the virtual laser pointer. When the virtual laser pointer is in a position that identifies a spatial point to be marked, a remote control or other input device (e.g., a keyboard, mouse, trackball, switch, etc.) may be activated to allow the medical professional to mark one or more spatial locations. For example, the medical professional may mark reference 1 and / or reference 2. Similarly, a purely virtual laser pointer may be controlled by a joystick, mouse, or other input pointing control device that is not within the field of view of the AR headset.

[0027] In one configuration, the location of an anatomical feature within a person's body can be identified using at least two fluoroscopic (i.e., X-ray) images from a fluoroscopic imaging device. The location of the anatomical feature within the person's body can be identified by a medical professional using a first fluoroscopic image. The fluoroscopic image is used to locate the anatomical feature (e.g., spinous process, iliac crest, top of the femur, etc.) because the medical professional cannot see through external tissue or skin. The anatomical feature can be marked in the fluoroscopic image. The tip of a physical pointer object can be inserted or immersed within the person's body and touch the anatomical feature. For example, the tip of the physical image pointer may touch a spinous process that is underneath other tissue. One fluoroscopic image can allow the anatomical feature to be identified in the X and Y coordinate directions, and then a second fluoroscopic image can be used to identify the anatomical feature in the Z direction. Thus, the tip of the physical pointer object can be positioned to touch a non-visible anatomical feature (visible only in the fluoroscopic image) as a referenced three-dimensional feature. The tip of a physical pointer object (e.g., a wand, forceps, a metal ball at the tip of a pointer, etc.) can then be registered as a spatial location. Additionally, image datasets can be registered with the person's body using the spatial locations recorded for non-visible anatomical features relative to virtual anatomical markers in the image dataset (e.g., MRI images, CT scans, etc.).

[0028] This allows the location of the tip of the physical pointer object to be identified even when the tip is within the patient's tissue. In another configuration, a virtual needle or pointer can be generated when the visible portion of the needle or pointer is outside the patient's body, but the tip of the needle is inside the patient's body and is not visible. An optical code on the physical pointer object can provide a location for the entire physical pointer object, so that the tip of the needle can be virtually projected (as an overlay simulating the tip of the needle within the patient) even though the end of the actual needle within the patient is not visible in the actual view through the AR headset. Thus, virtual anatomical markers can be marked on the image, and the location of the virtual needle tip can be used to align or merge the actual view (e.g., reality) of the patient with an image dataset (e.g., a captured 3D medical image).

[0029] This technology overcomes the registration problem without the issue of markers or fiducials that move during surgery. Specifically, markers or fiducials may be placed on a patient's body during surgery to assist in the registration of image datasets. However, when the patient's skin or other tissues are incised or cut during surgery, the markers or fiducials are likely to move due to loose skin, incisions, tissue retraction, or changes in patient position (e.g., moving between supine and prone positions), displacing the markers or fiducials. As a result, this technology helps overcome this problem that may exist in registration methods that use manual registration, matching the curvature of the skin using mesh triangles, using optical codes on the body's exterior, or various other tags on the body. In contrast, this technology can be used to identify a specific part of the patient's anatomy (e.g., one of the transverse processes or posterior spinous processes), allowing a medical professional to locate that spatial point. The identified spatial point can become a fiducial or fixed reference point for the patient's body. As described above, the image datasets may be registered to identified spatial points using virtual anatomical markers within the image datasets. In one example, a medical professional may mark normally exposed anatomical structures of the body (e.g., tip of the nose, tips of the ears, bony prominences of the tibia, temples of the head, skull shape, iliac crest, etc.). A physical wand may then be moved by the medical professional to touch spots, follow virtual contours, or track shapes. For example, a physical wand may be used to track a spatial region from the temple of the patient's head. In addition, the medical professional may mark anatomical structures exposed during surgery and skin incision. As a result, markings may occur on visible anatomical structures that are visible before surgery or on visible anatomical structures that are only visible after surgery has progressed.

[0030] To reiterate, virtual anatomical markers may be created in the image dataset to be aligned with the spatial points identified by the physical pointer object. The virtual anatomical markers may be considered fiducials created in the image dataset that may be aligned with selected spatial points. For example, a spatial point may be identified by pointing or touching it with a wand or finger, and the identified spatial point may then be aligned with the currently selected virtual anatomical marker. Thus, fiducials may be created in the image dataset (e.g., a 3D dataset) and physical reference points are identified on the body. This combination of aligned fiducials may create an anatomical fiducial reference system.

[0031] FIG. 2 illustrates a physical pointer object that may be a physical wand 200 or a physical pointer object. The physical wand 200 may have a shaft 202 with a pointer 204 (e.g., a tip) at the end of the shaft 202. The physical wand may include a handle 206 having a loop or other structure that is convenient for a human hand to hold, as in this example. However, the handle on the physical wand may be any shape that can be held by a human hand. For example, the handle 206 may be a ball, a knob, a rod perpendicular to the shaft, a single loop, or another handle type. The handle of the physical wand 200 may have an optical code 208 on the handle 206. Alternatively, the optical code 208 may be located on the shaft of the physical wand 200 or physical pointer object. The optical code may be identified by an AR headset using an optical camera, and the location of the optical code 208 may be assigned coordinates in the 3D space viewed by the AR headset. Once the location of the optical code is known, the location and orientation of the physical wand 208 is also known, since the relationship between the optical code 200 and the physical wand is predetermined. Additionally, identifying the location of the optical code 208 allows the handle 206, shaft 202, and then the tip of the pointer 204 or physical pointer object to be identified.

[0032] FIG. 3A illustrates a top view of an example augmented reality (AR) environment in which an image dataset 302 of a patient or person 304 may be aligned with a real view of the patient using a spatial location identified using a physical pointer object 306 in the field of view of an optical sensor in an AR headset. The image dataset 302 of a portion of the person's body may be acquired using a medical imaging device, and the image dataset may have virtual anatomical markers 310 for identifying anatomical features. In one example, the virtual anatomical markers 310 may be placed by a medical professional marking the image with a virtual marking tool (e.g., a graphical mark). For example, locations for the virtual anatomical markers 310 in the image dataset may be defined using a computing device or workstation controlled by the medical professional prior to a medical procedure. The virtual anatomical markers 310 in the image dataset may be virtual markers that are dots, lines, curves, crosshairs (see FIG. 3G), virtual markers, two-dimensional regions, or three-dimensional graphical objects. The medical professional or other individual may touch points, trace curves, or trace shapes that are used for alignment with corresponding shapes of virtual anatomical markers in the patient's image dataset 302. For example, a physical or virtual wand may point to one spot at a time, while the medical professional may trace a contour or shape that becomes a virtual landmark that can be aligned with the virtual anatomical marker 310 in the image dataset 302.

[0033] In one configuration, the physical pointer object 306 may have a reflective IR (infrared) marker on or near the pointer or tip of the physical pointer object 306. The AR headset may be capable of projecting IR light from an AR source in the AR headset toward the physical pointer object 306. The reflection of the IR light may provide the AR headset with a specific location defined by the IR reflector. Alternatively, the physical pointer object 306 may have an IR light source that can be registered by an IR sensor in the AR headset. The strength and position of the IR signal in the AR headset may be used to determine the location of the physical pointer object 306 and / or the tip or pointing structure of the physical pointer object 306.

[0034] The optical code 305 can also identify image visible markers below the optical code that are visible in the image dataset 302. The optical code 305 can be used to align the image dataset 302 with the patient's body, as described in more detail in U.S. Patent Application No. 16 / 194,333, entitled "Using Optical Codes with Augmented Reality Displays," filed November 17, 2019, which is incorporated herein by reference in its entirety.

[0035] FIG. 3B illustrates that multiple optical codes 308a and b may be used on the physical pointer object 306. Using multiple optical codes 308a and b may increase the accuracy of identifying the location of the physical pointer object and / or the tip, end, pointer, or pointing structure of the physical pointer object 306. The AR headset may capture the size and orientation of the light tags 308a and b. This allows the system (e.g., the AR headset) to make a better estimate of the orientation of the physical pointer object 306 (e.g., a wand) in the 3D space viewed by the AR headset. In one embodiment, a line in 3D space may be created through the center points of the light tags 308a and b attached to the physical pointer object 306 at the depth of the light tags 308a and b. Because the length of the physical pointer object 306 is known, the tip or point of the physical pointer object may be determined.

[0036] FIG. 3C illustrates that the accuracy of using a single physical pointer object to determine a spatial point can be further increased by orbiting, precessing, or rotating the physical pointer object 307 around the registered spatial point 310. Two or more spatial positions 312a-d of the physical pointer object 307 can be captured. The rotation of the physical pointer object can be rotated left or right, or can be rotated toward or away from the AR headset at any angle. The location of the physical pointer object 307 can be identified at the first position 307 using one or more optical codes 311. A virtual line can be created for each spatial orientation of the physical pointer object (e.g., multiple lines can be created for multiple spatial positions), and the lines can be used to identify the registered spatial point 310. The use of two or more optical codes can produce greater accuracy than only one optical code, as a line can be drawn between both points to assist in locating the tip of the physical pointer object. Additionally, the point depicted by the optical tag may become more accurate within the 3D space of the AR headset as the optical tag is moved left to right or up and down relative to the AR headset.

[0037] The physical pointer object may be moved to a second position 312a, and the virtual line 303 for the physical pointer object may again be determined. This movement of the physical pointer object may be repeated multiple times 312b-312d without moving the tip or point of the physical pointer object. A virtual line may be created for each spatial orientation of the physical pointer object. The virtual line may extend through the registered points for a defined distance, or the line may extend indefinitely through the registered points. Each time the physical pointer object 307 is rotated relative to where the physical pointer object is in contact on the human patient's body, a new virtual line may be calculated for that spatial position of the physical pointer object 307.

[0038] The multiple virtual lines may have virtual intersections or best-fit (e.g., approximate) intersections calculated for a group of lines. The virtual intersections or approximated virtual intersections may be considered spatial points that the medical professional wants to locate and that are aligned with the virtual anatomical markers 310. The intersections (or closest approximations of the intersections) of the virtual lines may then be calculated and then determined to be spatial points. Thus, spatial points may be more accurately identified using multiple positions of the physical pointer object and the virtual lines to identify the spatial points. Capturing multiple positions for the physical pointer object can overcome issues of variability in positioning the physical pointer object relative to the AR headset (e.g., it does not matter whether the physical pointer object is initially positioned too far away, too far to the right or left, etc.).

[0039] Although a more precise location can be determined using a virtual line generated when the optical tag is moved left and right or up and down relative to the AR headset, the optical tag 311 tends to be less precise in the depth direction from the AR headset. Detecting where an optical code is located in depth is driven in part by the size of the optical code, which has some limitations in terms of depth resolution. One reason for these limitations is that calculating the depth of an optical code from the AR headset may use the size of the tag, but if the image recognition software is off by a fraction of a pixel due to lighting or other issues, the position registration of the physical pointer object or wand tip may be off by millimeters, which can cause problems in medical or other engineering settings. While a spatial point can be identified using a physical pointer object at a single location or multiple locations using a virtual line, any errors introduced through the optical code, lighting, AR headset, optics, or computing system when identifying the optical code can result in a less accurate spatial point. As a result, a virtual plane (described below) can be used to correct for such depth issues.

[0040] FIG. 3D illustrates an example of a plane that can be formed using an optical code on a physical pointer object viewed from multiple angles or at multiple spatial locations and the location of a camera on an AR headset. To form a plane, at least three points are required in the 3D space of the AR headset. The at least three points can be accurately determined. One of these points is the 3D location of the camera 322 in the AR headset (e.g., the center of the medical professional's forehead). Additionally, while the 3D location of the optical code can be accurately determined in X, Y, and Z coordinates, the depth of the optical code location is less accurate in depth (e.g., in the Z direction). Therefore, a 3D plane 325a can be calculated using these three points. Even if the depth of a point from the optical code is not very accurate (off by millimeters), the plane 325a is still the same because the optical code point is still within the plane, but is simply farther away within the plane. Using a plane to find a spatial point is much more accurate than using only a line. The location of the camera 322 in the AR headset is known. The second two points of the plane are points on each of the two optical codes. The center (or other point) of the optical code can be used as two additional points in the plane 325 being created. The plane can extend infinitely from the plane created by the three points used to define the plane (although in FIG. 3D a finite triangular plane is used to illustrate the three points used to form the plane).

[0041] The medical professional creating the plane and / or holding the physical pointer object in contact with the patient 329 may be looking at the edge of the plane, and an infinite plane may pass through or bisect the medical professional in the direction of the edge. The medical professional may view the physical pointer object 328 from multiple angles, as exemplified by the second camera 327, to create multiple planes with optical codes on the physical pointer object. Additionally, the medical professional may move the physical pointer object 328 left and right through various degrees of rotation (e.g., 20-90 degrees different from the initial plane) to create multiple different planes. These planes may intersect at the pivot point of the physical pointer object, which may be on an anatomical feature of the patient 328 (e.g., on the skin, on the bone under surgery, or another anatomical feature).

[0042] 3D illustrates two other planes 325b-c that are created when a medical professional views the physical pointer object from a different perspective or when the physical pointer object is rotated around a point. Calculating the intersection of three or more planes 325a-c results in a spatial point that can be registered to a virtual anatomical marker in the image dataset with the patient's body. While at least three planes are used to calculate the spatial point, more planes may increase the accuracy of spatial point registration. Alternatively, two virtual lines created from the physical pointer object may identify the spatial point.

[0043] Using the planes of this technology, the spacing between optical codes and the length of the physical pointer object do not need to be known. Instead, three known points are formed on the plane, regardless of the distance between the points. A medical professional looking at the physical pointer object can create 20-30 planes, and then an intersection operation can be performed. The intersection operation can use an optimization calculation to calculate the plane intersection point (e.g., a usable spatial point) that is the shortest distance from all planes.

[0044] In one configuration, a spatial point can be found by first calculating the spatial point using one or two lines captured using a physical pointer object, which is a good initial measurement. Then, after a few minutes, when enough planes have been captured, a more accurate spatial point can be calculated and registered. The use of at least three or more planes oriented at substantial angles to each other increases the accuracy of the intersection. Planes that are only a few degrees apart are less accurate because such planes may be too close to each other parallel to provide a very accurate intersection. A more accurate intersection point can be calculated when planes separated by a relative angle of 20 to 90 degrees are used. Ideally, the planes intersect at a single point, but a best-fit calculation can be performed to determine the spatial point even when an ideal intersection is not available. In one embodiment, the measurement accuracy or registration of a spatial point in the 3D space viewed by the AR headset as found using the planes can be within a tenth of a millimeter or even less.

[0045] The use of planes helps correct for the inaccuracy of determining the depth of an optical code relative to an AR headset or in 3D space. The optical code may appear larger or smaller depending on the amount of light reflected from the optical code. This is because computer vision processes can use thresholds to find the boundary between black and white. If the room is brighter, brighter areas will overwhelm darker areas through a thresholding process. This can result in a fraction of a pixel of change in apparent size. As a result, the optical code may appear larger or smaller than one millimeter depending on how much light is hitting it. A one-millimeter difference in the size of the optical code may be registered to the system as a one-centimeter difference in depth, which can result in measurement inaccuracies in registering spatial points. Illumination on optical codes can be difficult to control in operating rooms or other scenarios, so relying on the depth of the code may not be very useful. However, the center of the optical tag can be calculated accurately and in an angle-independent manner by calculating a ray of light from the AR camera through the center of the optical tag. Using a plane to find the spatial point can correct for significant errors in the depth of the optical tag. Any optical tag moved along a ray from the AR camera will still be in the same plane defined by the location of the optical tag and the AR camera. Therefore, using a plane makes the spatial point calculation less sensitive to the depth of the optical code.

[0046] 3E illustrates a first virtual pointer plane 330. This first virtual pointer plane 330 can be created using the AR camera location on the physical pointer object 306 and one or more optical codes 332a and b. The plane from FIG. 3E is viewed from the edge, as seen from the perspective of the medical professional or the AR headset camera. The AR headset camera is at the top of FIG. 3E. The first virtual plane 330 can be created using points from the AR headset camera and points from the optical code, and the plane can pass through the optical code, the pointing structure 306 (e.g., a rod in this case), and the overall physical pointing object.

[0047] A second virtual plane 334 may be created when the physical pointer object 306 is rotated or orbited around the point 310 being touched by the physical pointer object 306. The intersection between three or more planes may be used to identify the point in space being selected by the medical professional. The physical pointing device 306 may be positioned at any number of rotational or orbital positions around the point on the patient 304 being touched by the medical professional, and a virtual pointer plane may be generated for each of the physical pointing device's multiple spatial positions. For example, the physical pointing device may be oriented at any number of different angles (any orientation between 0 and 360 degrees in multiple axes around the point) relative to the identified or selected point in space.

[0048] FIG. 3F illustrates an example of a physical pointing device 340 having two optical codes 342a and 342b rotated around a point, and multiple virtual pointer planes are generated using the known location of the camera in the AR headset and a point within each of the optical codes 342a and 342b. The multiple virtual pointer planes are then used to generate a localized or selected point in space 344. Three or more virtual pointer planes may intersect with each other at a localized or selected point in space, and the virtual pointer planes may be used to calculate the point in space 344. Plane generation may be triggered automatically (e.g., at timed intervals) as the physical pointing device is moved, or only upon explicit input from a medical professional when the physical pointing device is at a desired orientation. The virtual pointer plane may pass through the physical pointing object, away from the AR headset wearer, and extend infinitely upward and downward through the AR headset wearer to create a mathematical intersection of the planes. In other words, the plane may be a mathematically infinite plane, or the plane may be large enough to provide the necessary plane intersection.

[0049] Figure 3G illustrates an example of a physical pointing device 350 pointing to physical anatomical structures beneath the skin of a patient's body. Figure 3G further illustrates spatial points or internal reference points 352a-c (e.g., L5, T12, T8, T4) that have been registered using the physical pointing device 350. Figure 3H illustrates virtual anatomical markers or reference points marked on an image dataset (e.g., T8 and T12) and illustrated in a two-dimensional view of the image dataset. Figure 31 illustrates the virtual anatomical marker or reference point illustrated in Figure 3H (e.g., the posterior spinous process of T12) and marked in a three-dimensional view of the image dataset.

[0050] In one example of a method used with the elements described in FIGS. 3C-3I, identifying a spatial location using a virtual plane (or virtual line) may include locating a spatial point. In one operation, a first virtual plane may be generated for a physical pointer object in a first spatial orientation. The virtual plane may be an extension of the physical pointer object or a virtual object that aids in locating a spatial point. For example, the virtual plane may be a plane formed using points defined by a camera in an AR headset and points from an optical code on or associated with the physical pointer object.

[0051] Another operation may be to generate a second and a third virtual plane (or a second and a third virtual line) for the physical pointer device in a second spatial orientation. When at least three virtual planes (or two virtual lines) exist, the intersection of the first, second, and third virtual planes may be calculated. The intersection of the three virtual planes defines a spatial location. In the case of virtual lines, at least two lines are generally used to identify a spatial point. This process may also be performed using many virtual planes. For example, 20 spatial positions of the physical pointer object may be captured, and then the virtual intersection of the virtual planes for each spatial location may be calculated to find the spatial location to be registered. FIG. 4A illustrates a cross-sectional view of a location on a vertebra of a person's body where a virtual anatomical marker 410 has been created by a medical professional to identify a location in an image dataset 412a, which may be aligned with the spatial location identified by the physical pointer object. For example, the virtual anatomical marker 410 may be a green circle in the image dataset.

[0052] 4B illustrates a lateral view of locations within a person's spinal region where virtual anatomical markers 422 have been created by a medical professional to identify locations within an image dataset 412b that may be aligned with spatial locations identified by a physical pointer object. Additionally, a red circle 420 can be seen where a pedicle screw will track during a medical procedure.

[0053] 4C illustrates a posterior view of locations within a person's spinal region where virtual anatomical markers 430 have been created by a medical professional to identify locations within an image dataset 412c that can be aligned with spatial locations identified by a physical pointer object. Additional markers 432 are illustrated to indicate locations where alignment tools and pedicle screws will be applied during a medical procedure.

[0054] The process of registering the image dataset with the patient's body using spatial points identified by the physical pointer object can be a multi-step process. The image dataset, which is a 3D image or a multi-planar image, can be viewed on a workstation as a 3D image or a multi-planar image. The image dataset can be previously captured using a desired imaging modality (MRI, CT scan, etc.). A medical professional viewing the image dataset can select and mark one or more virtual anatomical markers (e.g., fiducial reference points). These virtual anatomical marks refer to anatomical structures that a physician can identify in a medical procedure (e.g., surgery). These markings in the 3D image set can be marked using points, circles, squares, 2D planes, virtual 3D objects, etc.

[0055] The image dataset can be viewed as a 3D image and becomes an image overlay in an AR headset. When viewed through the AR headset, the image dataset can be referred to as a holographic image, and the image dataset can be registered to the patient. Registration can be performed using virtual anatomical markers and spatial points defined by a physical pointer object. The image dataset can be projected onto the patient and viewed using the AR headset.

[0056] FIG. 5 illustrates identifying a spatial location identified by a physical pointer object 510. The physical pointer object 510 may have an optical code on the physical pointing object, as illustrated by a physical wand 512 against the side of a human body (a cadaver is shown). In this example, the spine may be exposed, and then a medical professional or observer may view the spinous processes. The medical professional may move the physical wand, needle, or physical pointer object, and the optical code on the physical wand may be registered by the AR headset. The optical code may be fixed to the physical wand. This registration of the physical pointer object allows the AR headset to identify where the pointer or tip of the wand or needle is located in 3D space in front of the AR headset. The spatial location identified using the physical pointer object may then be registered with virtual anatomical markers in the image dataset.

[0057] This technique may improve registration compared to existing conventional registration methods that use manual registration, skin or body contour registration, registration using an optical code on the patient, or image-visible markers associated with an optical code on the patient's body. In this registration technique, virtual anatomical markers are used as references (e.g., virtual tags) in the image dataset, and the virtual anatomical markers may be registered with points, curves, regions, or structures that can actually be seen through the AR headset. Spatial locations (e.g., physical spots) that are to be registered with the image dataset may be quickly specified with greater three-dimensional accuracy compared to conventional methods by using a physical wand, needle, trocar, virtual wand, virtual laser pointer, or another physical or virtual pointer.

[0058] 6 illustrates a method for coexisting an image dataset with a person's body using an augmented reality (AR) headset. One operation may be acquiring an image dataset of a portion of the person's body, as in block 610. The image dataset may include virtual anatomical markers for identifying anatomical features, and the image dataset may be captured by a medical imaging device. The virtual anatomical markers may be created or marked by a physician or another medical professional using a separate computing device or workstation.

[0059] Another operation may be identifying the spatial location of a physical pointer object within the field of view of the AR headset's optical sensor, as in block 604. An optical code on the physical pointer object may enable the AR headset to determine the spatial location of the physical pointer object. Alternatively, the location of the physical pointer object may be determined by detecting the edges of the physical pointer object using a known edge and object detection process. For example, the spatial location of the physical pointer object may be a three-dimensional (3D) location detected by the AR headset's optical sensor. The physical pointer object may be a physical wand, needle, trocar, or any physical object having a pointing tip or pointing edge. The physical pointer object may also be a person's or medical professional's finger identified by the AR headset.

[0060] The spatial location of a visible anatomical feature identified using the physical pointer object may be recorded. The spatial location may be identified using a pointer (e.g., tip) or a marked point on the physical pointer object. Position registration of the spatial location may be initiated based on an input (e.g., keyboard, mouse click, foot pedal activation, hand signal to an AR headset, voice command, or another input) that the pointer or tip of the physical pointer object is located over at least a portion of a visible anatomical feature on a person's body, as in block 606. The spatial location may be a point, a tracked curve, or other tracked shape (e.g., 2D or 3D).

[0061] The image dataset may be aligned with the person's body using the recorded spatial locations for the visible anatomical features relative to virtual anatomical markers in the image dataset, as in block 608. For example, the virtual anatomical markers may be set to be at the same 3D location as the spatial location or at any offset from the spatial location. Additionally, the image dataset may be visually overlaid on the person's body using an AR headset.

[0062] 7 illustrates a method for coexisting an image dataset with a human body using an augmented reality (AR) headset. The method may be implemented using at least one processor and at least one memory device including a data store for storing a plurality of data and instructions that, when executed, cause the system to perform the method. A first operation may be acquiring an image dataset of a portion of the human body, as in step 702. The image dataset may be captured by a medical imaging device.

[0063] Another operation is to identify virtual anatomical markers associated with anatomical features in the image dataset, as in block 704. The virtual anatomical markers may be defined by a medical professional prior to the medical procedure. Additionally, a physical pointer wand may be identified using an optical sensor in the AR headset, as in block 706. The physical pointer wand may have an optical code thereon to enable recognition of the physical pointer wand. The spatial location (e.g., spatial coordinates) of the tip or pointer of the physical pointer wand may be determined by referencing the optical code.

[0064] The spatial position of the physical pointer wand may be tracked using an optical sensor in the AR headset, as in block 708. An input that the tip of the physical pointer wand is located at a visual anatomical feature may be received, as in block 710. For example, a button may be pressed and a sensor (optical eye) may be triggered to indicate that the physical pointer wand is at a desired spatial location. The spatial location of the tip of the physical pointer wand at the visual anatomical feature may be recorded, as in block 712. The image dataset may then be aligned with the person's body using the recorded spatial location of the tip of the physical pointer wand relative to virtual anatomical markers in the image dataset, as in block 714. This alignment occurs in the AR headset for the viewer using the AR headset.

[0065] In one configuration, a virtual pointer or graphical point extension may extend from and be controlled by a physical pointer object, and the tip of the virtual pointer may define a spatial location. The virtual pointer may be graphically generated by an AR headset, and the virtual pointer may be aligned with and / or controlled by one or more axes or points of the physical pointer object. For example, the virtual pointer may be aligned with the longitudinal axis of the shaft of the physical pointer object.

[0066] 8 illustrates the use of a user's finger as a physical pointing object. A first action may be acquiring an image dataset of a portion of a person's body, as in block 802, where the image dataset is captured by a medical imaging device. Virtual anatomical markers associated with anatomical features within the image dataset may be identified, as in block 804. The virtual anatomical markers may be defined by a medical professional prior to (or even during) the medical procedure using a separate computing device.

[0067] A finger of a user of an augmented reality (AR) headset may be identified using an optical sensor of the AR headset, as in block 806. The position of the finger may be tracked using an optical sensor of the AR headset, as in block 808. Input that the finger is pointing at a visible anatomical feature may be received at the AR headset, as in block 810. Further, the spatial location of the finger at the visible anatomical feature may be recorded. The image dataset may then be aligned with the person's body using the recorded spatial location of the finger tip relative to a virtual anatomical marker in the image dataset.

[0068] As described with respect to a physical pointer object or wand, a virtual pointer may extend from a user's finger and be visible through an AR headset. The virtual pointer may be a graphical pointer extended from the finger by the AR headset. The tip of the virtual pointer (or the end of the graphical pointer) may be used to identify a spatial location. In one implementation, the finger may have an optical code on it to enable finger recognition.

[0069] As previously discussed, the ability to register an image dataset with anatomical structures within a patient's body is extremely useful when performing many tasks in medical procedures. If an image dataset does not line up closely with previously imaged anatomical structures or actual human tissue, the image dataset is of little use in the context of a medical procedure. For example, if an image dataset representing human tissue (e.g., an imaged hole in the skull) does not line up with actual human tissue (e.g., an actual hole in the skull), the image dataset is of little use in the context of a medical procedure. In some cases, a coarsely registered image set may be useful for simulation or training purposes, but is not useful for surgical procedures. In contrast, the present technique enables registration of image datasets for MRI, CT, or PET scan datasets at a resolution that can enable medical personnel to use the image dataset in medical procedures with high precision.

[0070] 9 illustrates an exemplary system that may be used to use an augmented reality (AR) headset to cause an image dataset to coexist with the body of a person being viewed through the AR headset. This system 900 may include a camera device 902, an augmented reality system 920, a display device 930, and multiple databases or data stores. System 900 may also include one or more processors, memory, file systems, communication units, operating systems, and user interfaces, which may be communicatively coupled together. In some embodiments, system 900 may be, for example, a desktop computer, a server computer, a laptop computer, a smartphone, a tablet computer, an embedded computer, an AR headset, a VR headset, a computing cluster, etc.

[0071] The camera device 902 may be configured to capture visual data. In one example, the camera device 902 may be used to capture visual data during a medical procedure or a medical examination. The visual data captured by the camera device 902 may include images of a person's body (or body parts), optical codes, medical implements (e.g., medical instruments, implants, and the like). The camera device 902 may transmit the captured optical data to the augmented reality system 920. The system may also include surface sensors, optical sensors, infrared sensors, lidar sensors, or other sensors to detect and assist in the actual view or mapping of the actual view detected by the AR system. Any object or surface may be detected for an operating room, a patient, a room, a physical shape, a medical implement, or any other physical surroundings or object.

[0072] Augmented reality system 920 may include an image processing engine 922, a fiducial and alignment module 924, an image generation module 926, and an enhanced display buffer 928. For example, image processing engine 922 may receive captured visible image data from camera device 902 and analyze the visible image data to identify one or more optical codes, objects, or people within the visible image data. A number of different techniques may be used to identify objects or people within the visible image data, including, but not limited to, feature extraction, segmentation, edge detection, and / or object detection.

[0073] The image processing engine 922 may identify optical codes that may be affixed to both the physical pointer object and / or the patient's body in the image. Once the image processing engine 922 identifies an optical code (e.g., AprilTag, 2D barcode, QR code, and others), the image processing unit 922 accesses the optical code database 946 to retrieve information associated with the optical code. Additionally, the optical code may be associated with a particular patient, a particular procedure, or a particular object.

[0074] In some embodiments, as described above, fiducial and registration module 924 may cooperate with image processing engine 922 to reference and register the spatial location identified by the physical pointing device and the image dataset relative to one another. Additionally, fiducial and registration module 924 may use optical code information in medical object database 944 to properly identify the size and shape, and identification, of the optical code. Once the position and orientation of the physical pointing object is determined, fiducial and registration controller 926 may register any associated radiology images in radiology image data 942 with the spatial location identified by the physical pointing object. In some examples, radiology images are received from radiology image database 942 based on patient records in patient record database 940.

[0075] The image generation module 926 can generate and display within the display device 930 graphical data, a virtual laser pointer associated with a physical pointing object, a virtual tool, a 3D surgical path, 3D coloring or shading of a mass or organ, or highlighting of a mass, organ, or target when overlaid on top of the patient's body. In some embodiments, this information can be loaded into an expanded display buffer 928. This information can then be transmitted to the display device 930 (e.g., an AR headset) for display to a user.

[0076] In one embodiment, patient database 940 includes multiple patient records. Each patient record may include one or more medical procedures performed on the patient. The patient records may also include notes, instructions, or plans for the medical procedures. The patient records may also be associated with one or more radiology images in radiology image database 942. In some embodiments, the radiology images include representations of virtual anatomical markers that enable fiducials and registration module 926 to properly register the image datasets with the patient's body using spatial locations identified by a physical pointing device. In one embodiment, medical object data 944 includes information describing medical items, including medical instruments, implants, and other objects.

[0077] In some embodiments, the augmented reality system may be located on a server, which may be any computer system or cluster of computers capable of functioning in conjunction with an AR headset or display device 930. In such embodiments, the server may be configured to communicate with the AR headset over a computer network to transmit image data to the AR headset or receive data from the AR headset.

[0078] 10 illustrates a computing device 1010 on which modules of the present technology may be implemented. A computing device 1010 is illustrated on which a high-level embodiment of the present technology may be implemented. The computing device 1010 may include one or more processors 1012 in communication with a memory device 1012. The computing device may include a local communication interface 1018 for components within the computing device. For example, the local communication interface may be a local data bus and / or any associated address or control bus as desired.

[0079] The memory device 1012 may include modules 1024 executable by the processor 1012 and data for the modules 1024. The modules 1024 may perform the functions described above. A data store 1022 may also be located within the memory device 1012 for storing data related to the modules 1024 and other applications, along with an operating system executable by the processor 1012.

[0080] Other applications may also be stored in the memory device 1012 and executable by the processor 1012. The components or modules discussed herein may be implemented in software using a high-level programming language that is compiled, interpreted, or executed using a hybrid approach.

[0081] A computing device may also have access to I / O (input / output) devices 1014 that are usable by the computing device. One example of an I / O device is an available display screen for displaying output from the computing device. Other known I / O devices may be used with the computing device, as desired. Networking devices 1016 and similar communication devices may be included in the computing device. The networking devices 1016 may be wired or wireless networking devices that connect to the Internet, a LAN, a WAN, or other computing networks.

[0082] Components or modules shown as stored in memory device 1012 may be executed by processor 1012. The term "executable" may refer to program files in a format that can be executed by processor 1012. For example, a program in a high-level language may be compiled into machine code in a format that can be loaded into a random access portion of memory device 1012 and executed by processor 1012, or source code may be loaded by another executable program and interpreted to generate instructions in a random access portion of memory to be executed by the processor. The executable program may be stored in any portion or component of memory device 1012. For example, memory device 1012 may be random access memory (RAM), read only memory (ROM), flash memory, solid state drive, memory card, hard drive, optical disk, floppy disk, magnetic tape, or any other memory component.

[0083] Processor 1012 may represent multiple processors, and memory 1012 may represent multiple memory units operating in parallel with the processing circuitry. This may provide parallel processing channels for processing and data within the system. Local interface 1018 may be used as a network to facilitate communication between any of the multiple processors and multiple memories. Local interface 1018 may use additional systems designed to coordinate communications, such as load balancing, bulk data transfer, and similar systems.

[0084] Some of the functional units described herein have been labeled as modules to more particularly emphasize their implementation independence. For example, a module may be implemented as a hardware circuit comprising custom VLSI circuits or gate arrays, off-the-shelf semiconductors such as logic chips, transistors, or other discrete components. A module may also be implemented in programmable hardware devices such as field programmable gate arrays, programmable array logic, programmable logic devices, etc.

[0085] Modules may also be implemented in software for execution by various types of processors. An identified module of executable code may comprise one or more blocks of computer instructions, which may be organized as, for example, an object, a procedure, or a function. Nevertheless, the executable files of an identified module need not be physically located together, but may comprise heterogeneous instructions stored in different locations that make up the module and that, when logically combined together, achieve the purpose stated for the module.

[0086] Indeed, a module of executable code may be a single instruction, or many instructions, and may even be distributed across several different code segments, among different programs and across several memory devices. Similarly, operational data may be identified and illustrated herein within modules and may be embodied in any suitable form and organized within any suitable type of data structure. Operational data may be collected as a single data set or distributed across different locations, including across different storage devices. Modules may be passive or active, including agents operable to perform a desired function.

[0087] The techniques described herein may be stored on computer-readable storage media, including volatile and nonvolatile, removable and non-removable media implemented in any technology for storage of information such as computer-readable instructions, data structures, program modules, or other data, including, but not limited to, RAM, ROM, EEPROM, flash memory or other memory technology, CD-ROM, digital versatile disk (DVD) or other optical storage devices, magnetic cassettes, magnetic tape, magnetic disk storage devices or other magnetic storage devices, or any other computer storage medium that can be used to store the desired information and the techniques described.

[0088] Devices described herein may also include communications or networking equipment and connections that allow the devices to communicate with other devices. A communications connection is an example of a communications medium. Communications media typically embodies computer-readable instructions, data structures, program modules, and other data in a modulated data signal, such as a carrier wave or other transport mechanism, and includes any information delivery media. A "modulated data signal" means a signal that has one or more of its characteristics set or changed in such a manner as to encode information in the signal. By way of example, and not limitation, communications media include wired media, such as a wired network or direct-wired connection, and wireless media, such as acoustic, radio frequency, infrared, and other wireless media. As used herein, the term computer-readable media includes communications media.

[0089] Reference will be made to the examples illustrated in the drawings, and specific language will be used herein to describe the same. It will nevertheless be understood that no limitation of the scope of the technology is thereby intended. Alterations and further modifications of the features illustrated herein, and additional applications of the embodiments illustrated herein that will occur to those skilled in the art in possession of this disclosure, should be considered within the scope of the present specification.

[0090] Furthermore, the described features, structures, or characteristics may be combined in any suitable manner in one or more embodiments. In the preceding description, numerous specific details, such as embodiments of various configurations, have been provided to provide a thorough understanding of embodiments of the described technology. However, those skilled in the art will recognize that the technology can be practiced without one or more of the specific details, or with other methods, components, devices, etc. In other instances, well-known structures or operations have not been shown or described in detail to avoid obscuring aspects of the technology.

[0091] Although the present subject matter has been described in language specific to structural features and / or operations, it is to be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features and operations described above. Rather, the specific features and operations described above are disclosed as exemplary forms of implementing the claims. Numerous modifications and alternative arrangements may be devised without departing from the spirit and scope of the described technology.

Claims

1. 1. A method for coexisting an image dataset with a human body using an augmented reality (AR) headset, comprising: acquiring the image dataset of a portion of the body of the person, the image dataset including virtual anatomical markers for identifying anatomical locations, the image dataset being captured by a medical imaging device; Identifying a spatial location using a physical pointer object within a field of view of an optical sensor of the AR headset; registering a spatial location of a visible anatomical feature identified using the physical pointer object based in part on input that the physical pointer object is located at a visible anatomical feature of the body of the person; and registering the image dataset with the body of the person using the recorded spatial locations of the visible anatomical features relative to the virtual anatomical markers in the image dataset.

2. The method of claim 1 , further comprising defining locations for the virtual anatomical markers in the image dataset using a computing device controlled by a medical professional.

3. The method of claim 1 , wherein the spatial location is at least one of a point, a curve, a tracked region, or a tracked three-dimensional (3D) shape.

4. The method of claim 1 , wherein a pointer or tip on the physical pointer object is used to identify the spatial location.

5. The method of claim 1 , wherein a virtual pointer extends from and is controlled by the physical pointer object, the tip of the virtual pointer defining the spatial location.

6. The method of claim 1 , further comprising using the AR headset to display the image dataset visually overlaid on the body of the person.

7. The method of claim 1 , wherein identifying the spatial location of the physical pointer object further comprises identifying a three-dimensional (3D) spatial location using the optical sensor of the AR headset.

8. The method of claim 7 , wherein the physical pointer object has an optical code thereon to enable the AR headset to determine the spatial location of the physical pointer object.

9. The method of claim 1 , wherein the physical pointer object is a physical wand, a needle, a trocar, or a physical object having a pointing tip.

10. The method of claim 9 , wherein the spatial location of the wand can be recognized by the AR headset using an optical code on the wand.

11. The method of claim 1 , wherein the spatial location of the tip of the physical pointer object is recognized by detecting an edge of the physical pointer object.

12. The method of claim 1 , wherein the physical pointer object is a finger of a medical professional identified by the AR headset.

13. The method of claim 9 , wherein a virtual pointer extends from a finger, a distal end of the virtual pointer is controlled by the finger, and the distal end of the virtual pointer represents the tip of the physical pointer object.

14. 10. The method of claim 1, further comprising using a fluoroscopic imaging device to identify a spatial location of a tip of a physical pointer object, the tip of the physical pointer object touching the visible anatomical feature.

15. identifying a location of a non-visible anatomical feature within the human body using at least two fluoroscopic images; inserting a tip of the physical pointer object into the body of the person to touch the anatomical feature; registering the location of the tip of the physical pointer object as the spatial location; registering the image dataset with the body of the person using the recorded spatial locations of the non-visible anatomical features relative to the virtual anatomical markers in the image dataset; The method of claim 1 further comprising:

16. The method of claim 1 , wherein the virtual anatomical marker in the image dataset is at least one of a circle, a line, a curve, a spline, a point, a crosshair, a plane, a callout, a two-dimensional graphical object, or a three-dimensional graphical object.

17. The method of claim 1 , further comprising automatically retrieving the image dataset for the body of the person using one or more optical codes on the body of the person.

18. The method of claim 1 , wherein the optical code of the physical pointer object includes data or values ​​embedded in the optical code.

19. The method of claim 1 , wherein the optical code of the physical pointer object includes embedded data defining at least one of a medical device type, a medical device model, a medical device version, or a medical device manufacturer.

20. The method of claim 1 , wherein the optical codes of the physical pointer objects include a unique code for each physical pointer object.

21. The method of claim 1 , wherein a value in the optical code of the physical pointer object is used to verify licensing for at least one of virtual medical procedure software, registration software, or navigation software.

22. 10. The method of claim 1, wherein the value in the optical code of the physical pointer object is used in conjunction with optical codes of additional medical instruments to verify licensing or payment for at least one of virtual medical procedure software, registration software, or navigation software.

23. Using a physical pointer object to identify a spatial location generating a first virtual plane for the physical pointer object in a first spatial orientation; generating a second virtual plane for the physical pointer object in a second spatial orientation; generating a third virtual plane for the physical pointer object in a third spatial orientation; 10. The method of claim 1, further comprising: calculating an intersection of the first virtual plane, the second virtual plane, and the third virtual plane, wherein the intersection of the first virtual plane, the second virtual plane, and the third virtual plane defines the spatial location.

24. Using a physical pointer object to identify a spatial location generating a first virtual line for the physical pointer object in a first spatial orientation; generating a second virtual line of the physical pointer object in a second spatial orientation; 2. The method of claim 1, further comprising: calculating an intersection of the first virtual line and the second virtual line, wherein the intersection of the first virtual line and the second virtual line defines the spatial location.

25. The method of claim 1 , further comprising an IR reflector on the physical pointer object, wherein the AR headset projects IR light that reflects from the IR reflector to determine the location of the physical pointer object.

26. The method of claim 1 , further comprising: an IR light source on the physical pointer object; and an IR sensor on the AR headset used to determine the location of the physical pointer object.

27. 1. A system for coexisting an image dataset with a human body using an augmented reality (AR) headset, comprising: at least one processor; and at least one memory device including a data store for storing a plurality of data and instructions, the instructions, when executed, causing the system to: acquiring the image dataset of the body portion of the person, the image dataset being captured by a medical imaging device; identifying virtual anatomical markers associated with anatomical features in the image dataset, the virtual anatomical markers being defined by a medical professional; Identifying a physical pointer wand using an optical sensor of the AR headset; and tracking a spatial position of the physical pointer wand using the optical sensor of the AR headset. receiving input that a pointer of the physical pointer wand is positioned over a visual anatomical feature; recording the spatial location of the tip of the physical pointer wand at the visual anatomical feature; and aligning the image dataset with the body of the person using the recorded spatial location of the tip of the physical pointer wand relative to the virtual anatomical marker in the image dataset.

28. 28. The system of claim 27, wherein a virtual pointer extends from and is controlled by the physical pointer wand, and the tip of the virtual pointer defines the spatial location.

29. 28. The system of claim 27, wherein the physical pointer wand has one or more optical codes thereon to enable recognition of the physical pointer wand.

30. 28. The system of claim 27, wherein the spatial location of the tip of the physical pointer wand is determined by referencing an optical code of the physical pointer wand.

31. 28. The system of claim 27, wherein the tip of the physical pointer wand is recognized by detecting at least a portion of an outline of the physical pointer wand.

32. A non-transitory machine-readable storage medium having embedded thereon instructions that, when executed by one or more processors, cause the one or more processors to perform a process that: acquiring an image dataset of a portion of a human body, the image dataset being captured by a medical imaging device; identifying virtual anatomical markers associated with anatomical features in the image dataset, the virtual anatomical markers being defined by a medical professional; Identifying a finger of a user of an augmented reality (AR) headset using an optical sensor of the AR headset; Tracking a position of the finger using the optical sensor of the AR headset and receiving input from the AR headset that the tip of the finger is pointing at a visible anatomical feature; and registering a spatial location of the finger with the visible anatomical features and aligning the image dataset with the body of the person using the spatial location recorded for the finger relative to the virtual anatomical markers in the image dataset.

33. 33. The non-transitory machine-readable storage medium of claim 32, wherein a virtual pointer is a graphical pointer extending from the user's finger and visible through the AR headset.

34. 33. The non-transitory machine-readable storage medium of claim 32, wherein the finger has an optical code thereon to enable recognition of a position of the finger.

Images