Use an optical code attached to the patient to align the patient's image data with the patient's actual view

Abstract

Using an optical code attached to the patient to calibrate image data of the patient with an actual view of the patient. In some embodiments, a method may include attaching an optical code to the patient, attaching a marker pattern to the patient, sensing the optical code attached to the patient and a position of the optical code in 3D space, accessing image data, calculating a position of the marker pattern in 3D space, registering a position of an inner layer of the patient by calibrating the calculated position of the marker pattern in 3D space with a position of the marker pattern in the image data, and displaying the inner layer of the patient derived from the image data projected on the actual view of the patient in real time in an augmented reality (AR) head mounted device.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application is a continuation of U.S. application No. 15 / 979,283 filed on May 14, 2018, which is incorporated herein by reference in its entirety. Background Art

[0003] Augmented reality (AR) systems typically take a user's real-time view of a real environment and augment that view with computer-generated virtual elements (e.g., video, sound, images, or graphics), thereby augmenting the user's current perception of reality.

[0004] A common problem AR systems face is accurately aligning the position of virtual elements with a live view of the real environment. Another common problem AR systems face is consistently retrieving the correct virtual elements that correspond to the live view of the real environment. These retrieval and alignment processes are often done manually, which can be time-consuming, tedious, and inaccurate.

[0005] The subject matter claimed herein is not limited to embodiments that solve the disadvantages described above or that operate only in the environment described above. Rather, this background section is provided merely to illustrate one exemplary technology area in which some embodiments described herein may be implemented. Summary of the invention

[0006] In some embodiments, a method for calibrating a patient's image data with an actual view of the patient using an optical code attached to the patient may include different actions. For example, the method may include attaching an optical code to the patient, wherein the optical code can be sensed by an optical sensor. The method may also include attaching a marker pattern to the patient at a fixed position relative to the position of the optical code, wherein the marker pattern can be sensed by a non-optical imaging mode. The method may further include capturing the patient's image data using a non-optical imaging mode, wherein the image data includes the patient's inner layer, and the image data further includes a marker pattern in a fixed position relative to the position of the patient's inner layer. The method may also include using an optical sensor of an augmented reality (AR) head-mounted device to sense the optical code attached to the patient and the position of the optical code in 3D space. The method may further include accessing image data based on the optical code. The method may further include calculating the position of the marker pattern in 3D space based on the sensed position of the optical code in 3D space and the fixed position of the marker pattern relative to the position of the optical code. The method may further include: based on the calculated position of the marker pattern in the 3D space and the fixed position of the marker pattern in the image data relative to the position of the patient's inner layer, registering the position of the patient's inner layer in the 3D space by calibrating the calculated position of the marker pattern in the 3D space with the position of the marker pattern in the image data. The method may also include: in the AR head-mounted device, displaying the patient's inner layer in real time based on the registration, derived from the image data projected on the actual view of the patient.

[0007] In some embodiments, attaching the optical code to the patient may include attaching a bandage printed with the optical code to an outer layer of the patient. In these embodiments, attaching the marking pattern to the patient at a fixed position relative to the optical code position may include attaching the marking pattern to the bandage at a fixed position relative to the optical code position. Also, in these embodiments, the marking pattern may be embedded within the bandage, such as embedding the marking pattern within the ink used to print the optical code on the bandage, and the ink includes a material that can be sensed by a non-optical imaging mode. In these embodiments, the material that can be sensed by the non-optical imaging mode can be a radiopaque material that is opaque to X-rays, a magnetically visible material, or a radioactive material.

[0008] In some embodiments, the non-optical imaging mode may include a magnetic resonance imaging (MRI) mode, a computed tomography (CT) scanning mode, an X-ray mode, a positron emission tomography (PET) mode, an ultrasound mode, a fluorescence mode, an infrared thermal imaging (IRT) mode, or a single photon emission computed tomography (SPECT) scanning mode.

[0009] In some embodiments, the image data may include two-dimensional (2D) image data, three-dimensional (3D) image data, four-dimensional (4D) image data, or some combination thereof.

[0010] In some embodiments, the optical code may be a linear barcode, a matrix two-dimensional (2D) barcode, a quick response (QR) code, or some combination thereof.

[0011] In some embodiments, the optical code may be linked to the patient's medical data, whereby the optical code may be used to access the patient's medical data. In these embodiments, the optical code may be a security credential linked to the patient's medical data, whereby the optical code may be used to access the patient's medical data without additional security credentials.

[0012] In some embodiments, affixing the optical code to the patient may include printing the optical code on the patient's skin.

[0013] In some embodiments, attaching the optical code to the patient may involve placing clothing printed with the optical code on the patient's body.

[0014] In some embodiments, an apparatus for calibrating image data of a patient with an actual view of the patient using an optical code attached to the patient may include a bandage, an optical code printed on the bandage, and a marker pattern attached to the bandage. The optical code is perceptible by an optical sensor. The marker pattern may have a fixed position in the bandage relative to the position of the optical code on the bandage. The marker pattern is perceptible by a non-optical imaging mode, whereby when image data of the patient is captured using the non-optical imaging mode, the image data includes the patient's inner layers, and the image data further includes the marker pattern at a fixed position relative to the position of the patient's inner layers.

[0015] In some embodiments, the marking pattern may be embedded in the ink used to print the optical code on the bandage. In these embodiments, the ink may include a material that can be sensed by a non-optical imaging mode. In these embodiments, the material that can be sensed by a non-optical imaging mode may be a radiopaque material that is opaque to X-rays, a magnetically visible material, or a radioactive material.

[0016] In some embodiments, the bandage may be made from a sterilizable material.

[0017] It should be understood that the above invention summary and the following detailed description are illustrative and do not limit the invention as claimed. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] The embodiments herein, along with additional features and details thereof, are described and illustrated through the use of the accompanying drawings, in which:

[0019] Figure 1 An exemplary augmented reality (AR) environment is shown in which an optical code affixed to the patient can be used to calibrate image data of the patient with the patient's actual view;

[0020] Figure 2 Shown is a Figure 1 Optical code;

[0021] Figure 3 An exemplary computer system that may be used in aligning image data of a patient with an actual view of the patient using an optical code affixed to the patient is shown;

[0022] Figure 4 is a flow chart of an example method for calibrating image data of a patient with an actual view of the patient using an optical code affixed to the patient;

[0023] Figure 5A-5B It is a photograph of the optical code and marking pattern attached to the patient;

[0024] Figure 5C-5D is a photo of the patient's image data, where Figure 5A-5B The marking pattern is visible in the image data; and

[0025] Figure 5E It is a real-time projection observed through an augmented reality (AR) headset. Figure 5A-5B The actual view of the patient Figure 5C-5D The image data of the view is a photo. DETAILED DESCRIPTION

[0026] Medical imaging processes can be used to create visual representations of the interior of a patient. More specifically, medical imaging processes can be used to reveal internal structures hidden by the patient's outer layers (e.g., the patient's skin or clothing) for various purposes (e.g., training, research, diagnosis, and treatment).

[0027] Conventional medical imaging systems create image data for a patient and then display that image data on a computer monitor. While viewing patient images on a computer monitor separate from the actual patient can be useful in training, research, diagnosis, and treatment, this separate viewing can also cause problems.

[0028] For example, if a surgeon needs to remove a tumor from inside a patient's brain, the surgeon may view an image of the patient's brain on a computer monitor. After viewing the location of the tumor on the computer monitor, the surgeon may shift his or her gaze from the computer monitor to the actual patient on the operating table and may attempt to identify the approximate location of the tumor inside the patient's brain on the actual patient. This method of identifying the approximate location of the tumor may be difficult and prone to error. For example, a surgeon may inadvertently identify the left side of the brain in an image as having a tumor when the tumor is actually on the right side of the brain. This error may cause the surgeon to mistakenly make an unnecessary incision on the left side of the patient's skull at the beginning of brain surgery, or cause the surgeon to mistakenly remove instruments from the tumor during brain surgery.

[0029] In another example, when a doctor needs to perform knee surgery on a patient, the doctor may view an image of the patient's knee on a computer monitor. After viewing the problem area of ​​the knee on the computer monitor, the doctor may shift his or her gaze from the computer monitor to the actual patient on the operating table, and may attempt to identify the problem area of ​​the knee on the actual patient for surgery. This method of identifying the problem area of ​​the knee may be difficult and prone to error. For example, the doctor may inadvertently pull up an image of the wrong patient on the computer monitor, not realizing that the patient on the operating table does not match the image on the computer monitor. This error may cause the surgeon to make an incision in the wrong location at the beginning of the knee surgery due to the natural variation of the problem area of ​​the knee from one patient to the next, or may cause the surgeon to mistakenly introduce instruments into the wrong internal area of ​​the knee during the knee surgery.

[0030] To avoid the problems that arise in the brain surgery and knee surgery examples described above, medical professionals may use an augmented reality (AR) headset to augment an actual view of a patient (e.g., a real-time view of a patient that can be viewed with the user's naked eyes or with the user's eyes through the lenses of an AR headset) with image data of the patient (e.g., one or more images that were previously captured and subsequently projected on a display (e.g., projected on the lenses of an AR headset)). In particular, the patient's image data may be calibrated or registered with the patient's actual view, and then an image derived from the image data may be projected on the patient's actual view in the AR headset. Unfortunately, however, accurate calibration or registration of the patient's image data with the patient's actual view is difficult to achieve because the calibration process is typically done manually, which can be time consuming, tedious, and inaccurate, and can retrieve incorrect image data for a given patient.

[0031] The automatic calibration process disclosed in U.S. Patent 9,892,564, which is incorporated herein by reference in its entirety, is one solution to the above-mentioned manual calibration problem. However, in some applications, such automatic calibration processes may be limited by the resolution of the mapping sensors in the AR headset and / or may be limited in situations where only a relatively small area of ​​the patient's skin or other outer layers is exposed.

[0032] Compared with traditional AR systems, the embodiments disclosed herein can provide various benefits. In particular, for example, the embodiments disclosed herein can use an optical code attached to a patient to calibrate the patient's image data with the patient's actual view. For example, a medical professional can attach an optical code (e.g., a QR code) and a marker pattern to a patient. Then, the medical professional can use a non-optical imaging mode to capture the patient's image data. The image data may include one or more inner layers of the patient and a marker pattern that is fixed relative to the position of the one or more inner layers of the patient. Then, the medical professional can use an AR head-mounted device to sense the optical code attached to the patient and the position of the optical code in 3D space. Then, the AR head-mounted device can automatically access the image data based on the optical code, and can automatically calculate the position of the marker pattern in 3D space based on the sensed position of the optical code in 3D space and the fixed position of the marker pattern relative to the optical code position. Then, the AR head mounted device can calibrate the calculated position of the marker pattern in 3D space with the position of the marker pattern in the image data based on the calculated position of the marker pattern in 3D space and the fixed position of the marker pattern in the image data relative to the position of the patient's inner layer, thereby automatically registering the position of the patient's inner layer in 3D space. Finally, the AR head mounted device can display one or more inner layers of the patient in real time from the image data projected on the actual view of the patient.

[0033] Thus, the embodiments disclosed herein enable a medical professional to view a virtual interior of a patient while viewing the actual patient through an AR head-mounted device, without having to perform any time-consuming, tedious, and inaccurate manual process of calibrating the image data with the actual view of the patient. Further, by automatically retrieving (e.g., during surgery) image data using the same optical code as used during the capture of the image data, it can be ensured that the image data retrieved through the AR head-mounted device matches the actual patient viewed through the AR head-mounted device without any time-consuming, tedious, and inaccurate manual process of retrieving the image data. When used for training, research, diagnosis, or treatment, these embodiments enable medical professionals to more easily and accurately locate target locations within a patient's body. Further, the embodiments disclosed herein enable medical professionals to more easily and accurately locate target locations within a patient's body using relatively small areas of the patient's skin or other outer layers (e.g., exposed areas) that are relatively small. Figure 2Automatic calibration is performed using a relatively small skin area of ​​the exposed patient 106).

[0034] As an example, when used in the brain surgery example discussed above, the embodiments disclosed herein can accurately align the patient's image data with the patient's actual view, which can then avoid the surgeon's confusion about the location of the tumor on the left or right side of the brain, and thereby avoid the surgeon from making unnecessary incisions on the wrong side of the skull at the beginning of the brain surgery. Further, the embodiments disclosed herein enable the surgeon to accurately guide instruments to the tumor internally during surgery, thereby removing the tumor. Similarly, when used in the knee surgery example described above, the embodiments disclosed herein can avoid the doctor using the wrong patient's image data, because the optical code that remains attached to the patient will be used by the AR head-mounted device to automatically retrieve the patient's image data that was previously captured with the same optical code attached to the patient at the beginning of the knee surgery. Further, the embodiments disclosed herein enable the surgeon to accurately guide instruments toward the desired internal area of ​​the knee during knee surgery.

[0035] Go to the attached picture, Figure 1 An example augmented reality (AR) environment 100 is shown in which an optical code 200 attached to a patient 106 can be used to calibrate image data of the patient 106 with an actual view of the patient 106. In some embodiments, the environment 100 can include a 3D space 102, a user 104, a patient 106, an AR head mounted device 108 that can communicate with a server 112 via a network 110, and the optical code 200. In some embodiments, the environment 100 can also include a virtual user interface 114, a virtual frame 116, an object 118, and a virtual cursor 122, all of which are displayed with dashed lines, thereby indicating that these virtual elements are generated by the AR head mounted device 108 and can only be seen by the user through the AR head mounted device 108.

[0036] In some embodiments, 3D space 102 may be any 3D space, including but not limited to a building room, such as an operating room with an operating table 103 (eg, Figure 1 In some embodiments, the 3D space 102 may be a space in which the user 104 can view the patient 106 while wearing the AR head mounted device 108.

[0037] In some embodiments, user 104 may be any user of AR headset 108, including but not limited to medical professionals (e.g., Figure 1As shown, the AR headset 108 can be used by a medical professional, an instructor, a researcher, a patient, or a caregiver of the patient. For example, a medical professional can use the AR headset 108 to perform a medical procedure on the patient 106, such as performing surgery on the patient 106. Similarly, a researcher or instructor can use the AR headset 108 when conducting medical research or guiding medical students. Furthermore, when a medical professional tries to explain a proposed medical procedure to the patient 106, the caregiver of the patient 106 or the patient 106 himself can use the AR headset 108.

[0038] In some embodiments, patient 106 can be any animal, which can be alive or dead, conscious or unconscious, intact or missing one or more body parts. For example, patient 106 can be a living adult (e.g., Figure 1 In another example, patient 106 may be a conscious animal being evaluated by a veterinarian to diagnose a medical condition. In another example, patient 106 may be a single limb or organ of a deceased person.

[0039] In some embodiments, the AR headset 108 may be any computer system in the form of an AR headset that can augment an actual view of the patient 106 with image data. For example, the user 104 may use the AR headset 108 to augment the actual view of the patient 106 with one or more inner layers of the patient 106, including but not limited to the skeleton 106b (e.g., Figure 1 In some embodiments, the AR headset 108 may perform such processing to augment the actual view of the patient 106 regardless of the current position of the user 104 in the 3D space 102. For example, the user 104 may move around the operating table 103 and may view the patient 106 from any angle within the 3D space 102, and the AR headset 108 may always continue to augment the actual view of the patient 106 with one or more inner layers of the patient 106, thereby allowing the user 104 to view the patient 106 and the image data of the patient 106 from any angle within the 3D space 102. The AR headset 108 may be combined with the above-described method described herein. Figure 4 The disclosed method 400 performs this process of augmenting the actual view of the patient 106 with image data. In some embodiments, the AR headset 108 may be a modified version of the Microsoft HoloLens.

[0040] In some embodiments, the network 110 can be configured to communicatively couple the AR head mounted device 108 and the server 112 or other computer system. In some embodiments, the network 110 can be any wired or wireless network or a combination of multiple networks configured to send and receive communications between the system and the device. In some embodiments, the network 110 may include a personal area network (PAN) (e.g., a Bluetooth network), a local area network (LAN) (e.g., a WiFi network), a metropolitan area network (MAN), a wide area network (WAN), or a storage area network (SAN). In some embodiments, the network 110 may also be coupled to or may include a portion of a telecommunications network to send data using a variety of different communication protocols (e.g., a cellular network).

[0041] In some embodiments, the server 112 may be any computer system capable of operating in conjunction with the AR head mounted device 108. In some embodiments, the server 112 may be configured to communicate with the AR head mounted device 108 in real time to convey image data to the AR head mounted device 108 or to receive data from the AR head mounted device 108. In addition, the server 112 may be used to offload some or all data storage or processing desired by the AR head mounted device 108.

[0042] In some embodiments, the virtual user interface 114 can be any virtual user interface generated by the AR head mounted device 108, which includes an option to change the display of one or more projected inner layers of the patient 106 of the image data from the patient 106. The virtual user interface 114 can further include other information that may be useful to the user 104. For example, the virtual user interface 114 may include real-time vital signs of the patient 106, such as heart rate, blood pressure, and respiratory rate. In another example, the virtual user interface 114 may include a stopwatch showing the amount of time that the patient 106 has been unconscious. In another example, the virtual user interface 114 may include the patient's 106 medical records or other medical data.

[0043] In some embodiments, the virtual frame 116 may be generated by the AR headset 108 to confine the projected interior of the patient 106 from the image data to within the volume of the virtual frame 116. As an example, the projected skeleton 106b of the patient 106 may be confined to Figure 1116 in the virtual frame 116. In some embodiments, the virtual frame 116 can also provide assistance to the user 104 by providing a reference frame for the user 104 when the user browses the projected image data. For example, the reference frame can provide assistance to the user when moving an axial slice, a coronal slice, a sagittal slice, or an oblique slice of the image data within the virtual frame 116. The slices can be two-dimensional (2D) slices and / or 3D slices. 3D slices can include curved slices, such as curved slices that follow the natural curves of anatomical features, or slices with depth as well as height and width. The user 104 can move these slices using gestures that require the user 104 to move their hands roughly in the direction of the lines of the virtual frame 116, whereby the display of the virtual frame 116 can make it easier for the user 104 to perform these hand movements.

[0044] In some embodiments, object 118 may be anything that user 104 wishes to insert into patient 106 through the outer layer of patient 106. For example, object 118 may include, but is not limited to, a scalpel (e.g., Figure 1 ), observation mirrors, drills, probes, other medical instruments, and even the hands of the user 104. Similar to registering the real-time position of the outer layer of the patient 106, the outer layer position of the object 118 can also be registered. However, unlike the patient 106 who can remain relatively still in the environment 100, the object 118 is likely to be frequently moved in the environment 100, so the real-time position of the object 108 can be automatically tracked in the 3D space 102 against the position of the outer layer of the registered patient 106. Then, if the user 104 inserts a portion of the object 118 into the outer layer of the patient 106, the AR head-mounted device 108 can display the virtual insertion position of the object 118 projected in the projected inner layer of the patient 106 derived from the image data. In this way, even if the actual insertion position of the object 118 is hidden from the actual view of the user 104, the virtual insertion position of the object 118 can be projected on the actual view of the user 104. Registration of the object 118 may be performed in a manner similar to the image data registration disclosed herein, where an optical code is affixed to the object 118 and the continuously updated position of the object 118 relative to the 3D space 102 is then determined by sensing the optical code via the AR headset 108 .

[0045] In some embodiments, the virtual cursor 122 may be a virtual cursor generated by the AR head mounted device 108 on the virtual user interface 114, on another virtual control, or at any other location in the 3D space 102. In some embodiments, the position of the virtual cursor 122 may correspond to the focus position 120 of the AR head mounted device 108, which may correspond to the position of the head of the user 104. The user 104 may use the virtual cursor 122 to select one or more options of the virtual user interface 114, which may sometimes be associated with one or more other actions of the user 104 (e.g., the user blinks) or one or more gestures of the user 104 (e.g., tapping two fingers simultaneously in the field of view of the AR head mounted device 108).

[0046] In some embodiments, the optical code 200 may be attached to the patient 106 before the image data of the patient 106 is generated, and then remain attached to the patient 106 while the user 104 views the patient 106 through the AR head-mounted device 108. In other embodiments, the optical code 200 may be attached to the patient 106 after the image data of the patient 106 is generated, and then remain attached to the patient 106 while the user 104 views the patient 106 through the AR head-mounted device 108. In either case, the AR head-mounted device 108 may then use the optical code 200 to automatically calibrate the image data of the patient 106 with the actual view of the patient 106. Further, when the image data is automatically retrieved using the same optical code 200 as used during the capture of the image data, doing so can ensure that the image data acquired by the AR head-mounted device 108 matches the actual patient 106 seen through the AR head-mounted device 108. This will be discussed in conjunction with the following. Figure 2 Additional aspects of the optical code 200 are discussed.

[0047] Modifications, additions, or omissions to the environment 100 are possible without departing from the scope of the present disclosure. For example, in some embodiments, there may be multiple users in the 3D space 102, each of which is wearing an AR head-mounted device 108, so as to simultaneously view the patient 106 enhanced with the image data of the patient 106. In another example, multiple patients may be simultaneously present in the 3D space 102 to allow the user 104 wearing the AR head-mounted device 108 to simultaneously view multiple patients enhanced with the image data of the patient. In another example, there may be multiple users wearing AR head-mounted devices 108 and multiple patients in the 3D space at the same time. In another example, a video of the view from the AR head-mounted device 108 may be captured by the AR head-mounted device 108 and then sent to a remote location, such as to a server 112 or to a remote AR head-mounted device or a virtual reality (VR) head-mounted device via a network 110, for viewing by other users. This example enables a remote user to guide a local user 104 through a medical procedure for the patient 106. Further, while environment 100 is generally disclosed as being in the context of a user 104 viewing a patient 106, it should be understood that environment 100 may be more broadly defined as any environment in which a user wishes to view one or more inner layers of any object, such as a tree, a rock, an oil field, or a planet.

[0048] Figure 2 Shown attached to Figure 1 Of the patients 106 Figure 1 Optical code 200. Reference Figure 1 and Figure 2, the optical code 200 can be sensed by an optical sensor (e.g., an optical sensor built into the AR head mounted device 108). In some embodiments, the optical code 200 can be a linear barcode, a matrix two-dimensional (2D) barcode, a quick response (QR) code, or some combination thereof. In some embodiments, the optical code 200 can be linked to the medical data of the patient 106 so that the medical data of the patient 106 can be accessed using the optical code 200. In these embodiments, the optical code can be a security credential linked to the medical data of the patient 106, whereby the optical code 200 can be used to access the medical data of the patient 106 without additional security credentials. In these embodiments, using the optical code 200 itself as a security credential can allow the user of the AR head mounted device 108 to access the medical images and other sensitive medical data of the patient 106 in a medical environment (e.g., during surgery on the patient 106) without having to manually or otherwise enter other security credentials by the user 104 before accessing the medical data of the patient 106. In other words, based on the assumption that any user who can directly access the optical code 200 is authorized to directly access the medical data of the patient 106 without any additional authentication, automatic sensing of the optical code 200 by the AR head-mounted device 108 can provide instant automatic access to medical image data and other medical data of the patient 106 without any additional work on the part of the user 104 of the AR head-mounted device.

[0049] The optical code 200 may be further associated with a marker 206 that is perceptible by a non-optical imaging modality. Examples of non-optical imaging modalities may include, but are not limited to, a magnetic resonance imaging (MRI) modality, a computed tomography (CT) scanning modality, an X-ray modality, a positron emission tomography (PET) modality, an ultrasound modality, a fluorescence modality, an infrared thermography (IRT) modality, or a single photon emission computed tomography (SPECT) scanning modality. By making the marker 206 with a material that is perceptible by a non-optical imaging modality, the marker 206 may be enabled to appear in any image data of the patient 106 captured using the non-optical imaging modality. Examples of the marker 206 include, but are not limited to, a metal sphere, a liquid sphere, a metal wire, and a metallic ink segment.

[0050] The marks 206 may be arranged in a pattern with fixed positions relative to the position of the optical code 200. For example, Figure 2In the disclosed embodiments, the optical code 200 may be printed on a bandage 202 (e.g., an adhesive bandage), and the marker 206 may be attached to the bandage 202 (e.g., by being embedded in the bandage 202, and thus not visible on any surface of the bandage). In this embodiment, the marker 206 may be arranged in a pattern having a fixed position relative to the position of the optical code 200 by arranging the marker 206 in a fixed pattern of the bandage 202. Additionally or alternatively, the marker 206 may be embedded within the optical code 200 itself, such as by embedding the marker 206 within an ink used to print at least some portion of the optical code 200 on the bandage 202, and the ink includes a material that can be sensed by a non-optical imaging mode, such as an ink that is not radiopaque and is therefore opaque to X-rays, an ink that is magnetically visible and thus visible on an MRI image, or an ink that is radioactive and thus visible on a PET image. In these embodiments, the optical code 200 itself may serve as both an optical code and a marking pattern. Additionally or alternatively, the markers 206 may be arranged in a manner that is not associated with the process of attaching the marker 206 to a bandage printed with the optical code 200, such as printing the optical code 200 on clothing (e.g., clothing 107) and attaching the markers 206 to the clothing article, or printing the optical code 200 directly on (at least temporarily) the skin 106a of the patient 106 and attaching the markers 206 directly on (at least temporarily) the skin 106a of the patient 106 or below the skin. In any of these embodiments, even if the markers 206 themselves are not visible or cannot be sensed by other sensors of the AR headset 108, by arranging the markers 206 in a pattern with fixed positions relative to the position of the optical code 200, the fixed positions can be used later to calculate the position of the pattern of markers 206 relative to the visible position of the optical code 200.

[0051] Still further, in some embodiments, one or more additional internal markers may be inserted into the patient 106. For example, one or more additional internal markers may be inserted into the breast of the patient 106 at the site of the biopsy mass so that the internal biopsy site can be tracked. One or more of these additional internal markers may then be located and triangulated using the pattern of markers 206 so that the internal biopsy site can be tracked. This may be particularly useful in situations where the part of the patient 106 body (e.g., the breast (which tends to move around depending on the position of the patient 106)) where the internal location is to be tracked is not very fixed relative to other parts of the patient's body.

[0052] Once the optical code 200 and the markers 206 are attached to the patient 106 in a fixed pattern, a medical professional or an automated system may use a non-optical imaging mode (which can perceive the markers 206) to capture image data of the patient 106 and the markers 206. In particular, the image data may include not only one or more inner layers of the patient 106 (e.g., bones 106b, muscles, organs, or fluids), but also a pattern of the markers 206 at fixed positions relative to the positions of the one or more inner layers of the patient 106. In other words, not only will the one or more inner layers of the patient 106 appear in the image data of the patient 106, but the markers 206 will also appear in the image data of the patient 106 in a fixed pattern, and the positions of this fixed pattern of the markers 206 will appear in the image data at positions fixed relative to the positions of the one or more inner layers of the patient 106. In one example, if the non-optical imaging mode is a CT scan mode, the CT scan image may display the bones 106b, organs, and soft tissues of the patient 106, and the markers 206 arranged at fixed positions relative to the positions of the bones 106b, organs, and soft tissues of the patient 106.

[0053] Once the image data of the patient 106 is captured, the optical code may remain attached to the patient for a period of time. During this period of time, the patient 106 may be moved, for example, from a medical imaging room of a hospital to an operating room of a hospital. In some embodiments, the bandage 202 may be made of a sterilizable material so that the bandage 202 may be attached to the patient 106 in a non-sterile environment (e.g., an X-ray room) and later in a sterile environment (e.g., an operating room), wherein the bandage 202 may be attached to the patient 106 but may need to be sterilized in the sterile environment. After this period of time has passed, the user 104 (e.g., a medical professional) may use the AR head mounted device 108 to determine the position of the optical code 200 in the 3D space 102. For example, the AR head mounted device 108 may include an optical sensor (e.g., a camera), which the AR head mounted device 108 may use to sense the optical code 200 attached to the patient 106 and the position of the optical code 200 in the 3D space 102. The AR head mounted device 108 can then access the image data of the patient 106 based on the optical code. For example, as described above, once the AR head mounted device 108 detects the presence of the optical code 200 in the 3D space 102, the AR head mounted device 108 can automatically retrieve the image data of the patient 106 without requiring any additional credentials from the user 104 of the AR head mounted device 108.

[0054] After sensing the presence of the optical code 200 in the 3D space 102, the AR head mounted device 108 can automatically calculate the position of the pattern of the marker 206 in the 3D space 102. This automatic calculation can be based on the sensed position of the optical code 200 in the 3D space 102, and can also be based on the known fixed position of the pattern of the marker 206 relative to the position of the optical code 200. In other words, even if the AR head mounted device 108 cannot perceive the marker 206 (for example, because the marker 206 is embedded inside the bandage 202), the AR head mounted device 108 can still automatically calculate the position of the pattern of the marker 206 based on the position of the optical code 200 sensed by the AR head mounted device 108 and based on the fixed position of the pattern of the marker 206 relative to the position of the optical code 200 known to the AR head mounted device 108. In this example, as long as the optical code 200 remains attached to the patient in the same position between the process of capturing image data and the process of sensing the optical code 200 by the AR head-mounted device 108, and as long as the positions of the pattern of markers 206 remain fixed relative to the position of the optical code 200, then these fixed positions enable the AR head-mounted device 108 to automatically calculate the positions of the pattern of markers 206 in the 3D space 102 even if the AR head-mounted device 108 cannot directly sense the positions of the markers 206 in the 3D space 102.

[0055] Then, after calculating the position of the pattern of markers 206 in the 3D space 102, based on the calculated position of the pattern of markers 206 in the 3D space 102 and the fixed position of the pattern of markers 206 in the image data relative to the position of one or more inner layers of the patient 106, the AR head mounted device 108 can calibrate the calculated position of the pattern of markers 206 in the 3D space 102 with the position of the pattern of markers 206 in the image data, thereby registering the position of one or more inner layers of the patient 106 in the 3D space 102. Thus, this calibration and registration process enables the AR head mounted device 108 to display one or more inner layers of the patient 106 in real time from the image data projected on the actual view of the patient 106.

[0056] Thus, the AR head mounted device 108 can use the optical code 200 and the pattern of associated markers 206 to automatically calibrate the image data of the patient 106 with the actual view of the patient 106. Furthermore, by automatically retrieving the image data using the same optical code 200 as used during the capture of the image data, it can be ensured that the image data retrieved by the AR head mounted device 108 matches the actual patient 106 viewed through the AR head mounted device 108.

[0057] Without departing from the scope of the present disclosure, modifications, additions, or omissions of the optical code 200 attached to the patient 106 are possible. For example, in some embodiments, multiple optical codes 200 may be attached to the patient 106 at the same time to further ensure accurate calibration of the image data of the patient 106 with the actual view of the patient 106 in the 3D space 102. In these embodiments, three or more optical codes 200 can achieve triangulation based on multiple optical codes 200. In these embodiments, the accuracy of triangulation can be improved by spacing the multiple optical codes 200 apart (e.g., a few inches apart). And, in some embodiments, Figure 2 The pattern of five markers 206 disclosed in the embodiment of the present invention may be replaced with another pattern (e.g., a three marker pattern or a seven marker pattern). Further, in some embodiments, because the markers 206 are attached to the outer layer of the patient 106, and because the outer layer of the patient 106 is not necessarily planar, the markers 206 are not necessarily all located on a single plane, but may be curved around any curvature of the outer layer of the patient 106. In these embodiments, the fixed position of the pattern of markers 206 relative to the position of the optical code 200 may be determined after the optical code 200 and the markers 206 are attached to the patient 106, because the fixed position may change according to the curvature of the outer layer of the patient to which the optical code 200 and the markers 206 are attached.

[0058] Figure 5A-5B is a photo of the optical code and the marking pattern attached to the patient. Figure 5C-5D is a photo of the patient's image data, where Figure 5A-5B The marking pattern is visible in the image data, and Figure 5E It is a real-time projection seen through an augmented reality (AR) headset. Figure 5A-5B The actual view of the patient Figure 5C-5D The image data of the view is a photo. Figure 5A-5B As disclosed in , a bandage can be attached to a patient, an optical code (e.g., a QR code) is printed on the bandage, and a marking pattern (e.g., four metal spheres) is attached to the bandage. The optical code can be sensed by an optical sensor. The marking pattern can have a fixed position in the bandage relative to the position of the optical code on the bandage. The marking pattern can be sensed by a non-optical imaging mode (e.g., a CT mode). Then, as Figure 5C-5D As disclosed in , when capturing image data (e.g., a CT image) of a patient using a non-optical imaging mode (e.g., a CT mode), the image data may include an inner layer of the patient (e.g., a bone of the patient), and the image data may further include a pattern of markers (e.g., four metal balls) at fixed positions relative to the positions of the inner layer of the patient. Later, a user may wear an AR headset (e.g., a surgeon may wear the AR headset during surgery on a patient). Figure 1 The AR head mounted device 108 is provided, and the optical sensor of the AR head mounted device can sense the optical code attached to the patient and the position of the optical code in the 3D space (e.g., the operating room). The AR head mounted device can then access the patient based on the optical code. Figure 5C-5D Then, the AR head-mounted device can calculate the position of the marker pattern in 3D space based on the sensed position of the optical code in 3D space and the fixed position of the marker (e.g., four metal balls) pattern relative to the position of the optical code. Then, the AR head-mounted device can calculate the position of the marker pattern in 3D space based on the calculated position of the marker (e.g., four metal balls) pattern in 3D space and the fixed position of the marker pattern in the image data (e.g., Figure 5C-5D The calculated position of the marker pattern in 3D space is calibrated with the position of the marker pattern in the image data by using a fixed position relative to the position of the patient's inner layer (e.g., the patient's bone) in the CT image, thereby registering the position of the patient's inner layer in 3D space. Figure 5E As disclosed in the AR head mounted device, the AR head mounted device can display in real time based on the registration in the AR head mounted device. Figure 5E A view shown in a photograph of a patient, wherein the view is derived from image data projected onto an actual view of the patient (e.g. Figure 5C-5D CT images of a patient's inner layers (e.g., the patient's bones).

[0059] Without departing from the scope of the present disclosure, Figure 5A-5B The optical code and marking pattern disclosed in Figure 5C-5D Modification, supplementation or omission of the image data disclosed in is possible. For example, Figure 5A-5B The marking pattern of the present invention may be replaced by other marking patterns with different numbers and / or different types of markings, instead of a marking pattern including four metal balls, thereby Figure 5C-5D The image data may include other marking patterns.

[0060] Figure 3 An exemplary computer system 300 is shown that can be used in a process of aligning image data of a patient with an actual view of the patient using an optical code attached to the patient. In some embodiments, the computer system 300 can be part of any system or device described in the present disclosure. For example, the computer system 300 can be Figure 1 part of either the AR head mounted device 108 or the server 112 .

[0061] The computer system 300 may include a processor 302, a memory 304, a file system 306, a communication unit 308, an operating system 310, a user interface 312, and an AR module 314 that can be communicatively coupled. In some embodiments, the computer system 300 may be, for example, a desktop computer, a client computer, a server computer, a mobile phone, a laptop computer, a smart phone, a smart watch, a tablet computer, a portable music player, an embedded computer, an AR head-mounted device, a VR head-mounted device, or any other computer system.

[0062] Generally, the processor 302 may include any suitable special or general-purpose computer, computing entity, or processing device that includes various computer hardware or software modules and can be configured to execute instructions stored on any applicable computer-readable storage medium. For example, the processor 302 may include a microprocessor, a microcontroller, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or any other digital or analog circuit configured to interpret and / or execute program instructions and / or process data, or any combination thereof. In some embodiments, the processor 302 may interpret and / or execute program instructions and / or process data stored in the memory 304 and / or the file system 306. In some embodiments, the processor 302 may obtain program instructions from the file system 306 and may load the program instructions into the memory 304. After loading the program instructions into the memory 304, the processor 302 may execute the program instructions. In some embodiments, these instructions may include being executed by the processor 302 Figure 4 One or more actions of method 400.

[0063] Memory 304 and file system 306 may include computer-readable storage media for carrying or storing computer-executable instructions or data structures. Such computer-readable storage media may be any available non-transitory media that can be accessed by a general-purpose or special-purpose computer (e.g., processor 302). By way of example and not limitation, such computer-readable storage media may include non-transitory computer-readable storage media, including read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), compact disc read-only memory (CD-ROM) or other optical disk storage devices, magnetic disk storage devices or other magnetic storage devices, flash memory devices (e.g., solid-state storage devices), or any other storage medium that can be used to carry or store desired program code in the form of computer-executable instructions or data structures and that can be accessed by a general-purpose or special-purpose computer. Combinations of the above may also be included within the scope of computer-readable storage media. As an example, computer-executable instructions may include instructions configured to cause processor 302 to perform an operation or a group of operations (e.g., Figure 4 The computer executable instructions may include one or more actions of the method 400 and instructions and data. As an example, these computer executable instructions may be included in the operating system 310, one or more applications (such as the AR module 314), or some combination thereof.

[0064] The communication unit 308 may include a device configured to communicate with the user through a network (e.g. Figure 1 The communication unit 308 may be any component, device, system, or combination thereof that transmits or receives information through the network 110 of the present invention. In some embodiments, the communication unit 308 may communicate with other devices located at other locations, the same location, or even other components within the same system. For example, the communication unit 308 may include a modem, a network card (wireless or wired), an infrared communication device, a wireless communication device (e.g., an antenna), and / or a chipset (e.g., a Bluetooth device, an 802.6 device (e.g., a metropolitan area network (MAN)), a WiFi device, a WiMax device, a cellular communication device, etc.), etc. The communication unit 308 may allow data to be exchanged with a network and / or any other device or system (e.g., a device or system described in the present disclosure).

[0065] Operating system 310 may be configured to manage hardware and software resources of computer system 300 , and may be configured to provide general services for computer system 300 .

[0066] The user interface 312 may include any device configured to allow a user to interact with the computer system 300. For example, the user interface 312 may include a display configured to present video, text, application user interface, and other data as directed by the processor 302, such as an LCD, LED, or other display (e.g., an AR lens). The user interface 312 may further include a mouse, a trackpad, a keyboard, a touch screen, a volume control, other buttons, a speaker, a microphone, a camera, any peripheral device, or other input or output device. The user interface 312 may receive input from a user and may provide the input to the processor 302. Likewise, the user interface 312 may present output to the user.

[0067] The AR module 314 may be one or more computer-readable instructions stored on one or more non-transitory computer-readable media (e.g., the memory 304 or the file system 306), which when executed by the processor 302 are configured to perform one or more methods (e.g., Figure 4 In some embodiments, the AR module 314 may be part of the operating system 310, or may be part of an application of the computer system 300, or some combination thereof.

[0068] Without departing from the scope of the present disclosure, modifications, additions, or omissions to the computer system 300 are possible. Figure 3 300, each component is illustrated as a single component, but any component 302-314 of the computer system 300 may include multiple similar components that function together and are communicatively coupled together. In addition, although illustrated as a single computer system, it should be understood that the computer system 300 may include multiple physical or virtual computer systems networked together (e.g., in a cloud computing environment, a multi-tenant environment, or a virtualized environment).

[0069] Figure 4 is a flow chart of an exemplary method for calibrating image data of a patient with an actual view of the patient using an optical code attached to the patient. In some embodiments, method 400 may be performed by a device or system, such as by a Figure 1 The AR head mounted device 108 and / or the server 112 running Figure 3 In these and other embodiments, the method 400 may be performed by one or more processors based on one or more computer-readable instructions stored on one or more non-transitory computer-readable media. Figure 1 , 2 , 3 and 4 to describe the method 400. Although various actions of the method 400 are described below as being performed by an AR head mounted device, it should be understood that these actions of the method 400 may alternatively be performed by another computer system or a combination of computer systems.

[0070] Method 400 may include, at action 402, attaching an optical code to a patient. In some embodiments, the optical code may be sensed by an optical sensor. In some embodiments, attaching the optical code to the patient may include attaching a bandage printed with the optical code to an outer layer of the patient. In some embodiments, the optical code may be a linear barcode, a matrix two-dimensional (2D) barcode, a quick response (QR) code, or some combination thereof. For example, at action 402, a medical professional or an automated system may attach the optical code 200 (in the form of a QR code) to the patient 106 by attaching a bandage 202 printed with the optical code 200 to the patient.

[0071] Method 400 may include, at act 404, attaching a marking pattern to the patient at a fixed position relative to the position of the optical code. In some embodiments, the marking pattern may be sensed by a non-optical imaging mode. In some embodiments where the optical code is printed on a bandage, attaching the marking pattern to the patient at a fixed position relative to the position of the optical code may include attaching the marking pattern to the bandage at a fixed position relative to the position of the optical code. In these embodiments, the marking pattern may be embedded within the bandage, such as embedding the marking pattern in an ink used to print the optical code on the bandage, and the ink includes a material sensed by a non-optical imaging mode. In these embodiments, the material sensed by the non-optical imaging mode may be a radiopaque material that is opaque to X-rays, a magnetically visible material, or a radioactive material. In some embodiments, attaching the optical code to the patient may include printing the optical code on the patient's skin. In some embodiments, attaching the optical code to the patient may include placing an article of clothing printed with the optical code on the patient. For example, at act 404, a medical professional or an automated system may attach a bandage printed with the optical code 200 and having a pattern of marking 206 attached to the patient 106. Figure 2 The fixed position relative to the position of the optical code 200 disclosed in Figure 2 The pattern disclosed in the invention can be used to attach the mark 206 to the patient 106. Alternatively, the pattern of the mark 206 can be directly embedded in the ink used to print the optical code on the bandage 202. Alternatively, the optical code 200 can be printed on the clothing 107, and the mark 206 can be attached to the clothing 107 placed on the patient 106; or the optical code 200 can be printed directly on the skin 106a of the patient 106 (at least temporarily), and the mark 206 can be directly attached to the skin 106a of the patient 106 (at least temporarily).

[0072] Method 400 may include, at act 406, capturing image data of a patient. In some embodiments, the image data of a patient may be captured using a non-optical imaging mode that can sense markers. In some embodiments, the image data may include inner layers of the patient and a pattern of markers that are in fixed positions relative to the positions of the inner layers of the patient. In some embodiments, the image data may include two-dimensional (2D) image data, three-dimensional (3D) image data, four-dimensional (4D) image data, or some combination thereof. For example, at act 406, a medical professional or an automated system may capture image data of a patient 106 using a non-optical imaging mode, the image data simultaneously including the inner layers of the patient (e.g., the bone 106 b of the patient 106) and a pattern of markers 206 that are in fixed positions relative to the positions of the bone 106 b of the patient 106.

[0073] In some embodiments, by using one or more methods, image data of patient 106 may be captured or generated in real time while patient 106 is in environment 100, and / or image data of patient 106 may be captured or generated before patient 106 enters environment 100. For example, some image data may be acquired before patient 106 enters environment 100, and then the image data may be enhanced with additional image data acquired in real time while patient 106 is in environment 100. As an example, the image data of patient 106 may include, but is not limited to, magnetic resonance imaging (MRI) images, computed tomography (CT) scan images, X-ray images, positron emission tomography (PET) images, ultrasound images, fluorescence images, infrared thermography (IRT) images, or single photon emission computed tomography (SPECT) scan images, or some combination thereof. Any of these images may be in the form of a still image or a video image. For example, method 400 may use a still X-ray image of the skeletal system of patient 106 (e.g., Figure 1 ). In another example, method 400 can use an ultrasound video image of a beating heart of patient 106. In another example, method 400 can switch between a still image of the heart and a real-time video of the heart beating.

[0074] Although the image data of the patient is acquired using a variety of different methods, in some embodiments, the image data of the patient may include the outer layer of the patient and multiple inner layers of the patient. As an example, the outer layer of the patient 106 may include Figure 1 In another example, the outer layer of the corpse may be a layer of tissue other than skin, such as a muscle or fat layer, wherein the skin has been removed from the corpse. The inner layer of the patient 106 may include, but is not limited to, the internal skeleton 106b of the patient 106 (e.g., Figure 1As shown), muscles, organs or body fluids. The image data may include 2D images (such as X-ray images) because the 2D images will have 3D meaning when projected into 3D space. As an example, the image data of the patient 106 may include a 2D X-ray image that can be projected onto the skin 106a or clothing 107 of the patient 106. The image data may also include a time element, which is sometimes referred to as four-dimensional (4D) data. As an example, the image data of one or more inner layers of the patient may include slices, transparent views, segmented images, or annotations. For example, the image data may include a video that not only includes a 3D image, but also includes a 3D image that changes over time. Multiple inner layers may be layers that run through the patient 106 or layers that extend only to a specific local depth of the patient 106. As an example, some forms of image data (such as 3D data obtained from a millimeter wave scanner) can only be configured to reveal items contained between the patient's outer clothing and skin. The image data may also be a combination of various types of image data.

[0075] A period of time may pass between actions 406 and 408 of method 400. During this period of time, the patient may be moved, such as from one part of a room to another part, or from one room of a hospital to another room of a hospital. In some embodiments, after moving patient 106 in this way, it may be necessary to take care to position patient 106 on operating table 103 in the same relative position and / or orientation as patient 106 when image data was captured. Alternatively, when patient 106 is positioned on operating table 103 in a different position and / or orientation than when image data was captured or generated, AR head mounted device 108 may deform the image data to match the different position and / or orientation of patient 106.

[0076] The method 400 may include, at act 408, sensing an optical code attached to the patient and a location of the optical code in 3D space. In some embodiments, the optical code may be sensed using an optical sensor of an AR head mounted device. For example, at act 408, an optical sensor of the AR head mounted device 108 may sense an optical code 200 attached to the patient 106 and a location of the optical code 200 in 3D space 102.

[0077] Method 400 may include: in action 410, accessing image data. In some embodiments, the image data may be accessed based on an optical code. In some embodiments, the optical code may be linked to the patient's medical data so that the patient's medical data may be accessed using the optical code. In these embodiments, the optical code may be a security credential linked to the patient's medical data so that the patient's medical data may be accessed using the optical code without additional security credentials. For example, in action 410, the AR head mounted device 108 may access the image data of the patient 106 based on the optical code 200 sensed in action 408, wherein the optical code 200 acts as a security credential linked to the medical data of the patient 106. Thus, the medical data of the patient 106 may be accessed using only the optical code 200 without additional security credentials.

[0078] The method 400 may include, at act 412, calculating a position of the marker pattern in 3D space. In some embodiments, the calculation of act 412 may be based on the sensed position of the optical code in 3D space and a fixed position of the marker pattern relative to the position of the optical code. For example, at act 412, the AR head mounted device 108 may calculate the position of the pattern of the marker 206 in 3D space 102 based on the sensed position of the optical code 200 in 3D space 102 and a known (e.g., previously determined) fixed position of the sensed pattern of the marker 206 relative to the position of the optical code 200.

[0079] Method 400 may include, at action 414, registering the position of the patient's inner layer in 3D space by calibrating the calculated position of the marker pattern in 3D space with the position of the marker pattern in the image data. In some embodiments, the registration of action 414 may be based on the calculated position of the marker pattern in 3D space and the fixed position of the marker pattern in the image data relative to the position of the patient's inner layer. For example, at action 414, based on the calculated position of the pattern of marker 206 in 3D space 102 and the known (e.g., previously determined) fixed position of the pattern of marker 206 in the image data relative to the position of the bone 106 b of the patient 106, the AR head mounted device 108 may register the position of the bone 106 b of the patient 106 in 3D space 102 by calibrating the calculated position of the pattern of marker 206 in 3D space 102 with the position of the pattern of marker 206 in the image data of the patient 106.

[0080] The method 400 may include: in action 416, displaying the inner layer of the patient in real time from the image data projected on the actual view of the patient. In some embodiments, the display of action 416 may be performed in the AR head mounted device based on the registration of action 414. For example, in action 416, the AR head mounted device 108 may display the bone 106b of the patient 106 in real time based on the image data projected on the actual view of the patient 106 based on the registration of action 414.

[0081] In some embodiments, act 416 of method 400 may further include repeating act 406 one or more times or continuously, and then displaying the newly captured image data resulting from the repeated performance of act 406 and the initial captured image data resulting from the initial performance of act 406, or displaying the newly captured image data overlaid on the initial captured image data. For example, one imaging mode, such as MRI, may be used during the initial performance of act 406, and then another imaging mode, such as fluorescence mode, may be used during the repeated or continuous performance of act 406 (e.g., during surgery to insert a catheter into a blood vessel of patient 106). Then, in act 416, the AR head mounted device 108 may display the initially captured MRI image overlapped with the newly captured fluorescence image projected on the actual view of patient 106 in real time based on the registration of act 414. In act 416, this overlap of the initially captured image data with the newly captured image data may be further accomplished using additional markers. For example, the fluoroscope may have one or more additional markings that calibrate the fluorescent image with the MRI image, thereby allowing the surgeon to more accurately locate the correct position for placing a catheter in the patient's 106 blood vessel and to more accurately guide the catheter internally into the patient 106 during surgery.

[0082] In some embodiments, the method 400 may use the optical code 200 affixed to the patient 106 to perform calibration of the image data of the patient 106 with the actual view of the patient 106. Further, such calibration may enable the medical professional to view the virtual interior of the patient 106 while viewing the actual patient 106 through the AR head mounted device 108 without performing the time-consuming, tedious, and inaccurate manual calibration of the image data with the actual view of the patient required by conventional calibration forms. Further, by automatically retrieving the image data using the same optical code 200 as used during the capture of the image data, it may be ensured that the image data retrieved by the AR head mounted device matches the actual patient as viewed through the AR head mounted device without performing the time-consuming, tedious, and inaccurate manual retrieval of the image data required by conventional calibration forms.

[0083] Despite Figure 4400 are illustrated as separate actions, but different actions may be divided into additional actions, combined into fewer actions, or reordered, expanded, or eliminated, depending on the desired implementation. For example, in some embodiments, actions 408-416 may be performed without performing any of actions 402-406, may be performed after previously performing actions 402-406, or may be performed by another entity different from the entity performing actions 402-416. Further, in some embodiments, only actions 408-414 may be performed, without performing any of actions 402-406 and 416. Furthermore, in some embodiments, action 412 may be modified to include sensing the position of a marker pattern in 3D space, rather than calculating the position of a marker pattern in 3D space (e.g., where the AR head mounted device 108 is capable of directly sensing the marker 206).

[0084] Furthermore, it should be understood that the method 400 can improve the functionality of the AR system itself and can advance the field of AR. For example, by using the optical code 200 attached to the patient 106 to calibrate the image data of the patient 106 with the actual view of the patient 106, the method 400 can be used to calibrate the image data of the patient 106 with the actual view of the patient 106. Figure 1 The functionality of the AR head mounted device 108 itself is improved. Compared with conventional AR systems that use manual registration processing, this automatic calibration process can be performed in a much easier and more accurate manner.

[0085] As indicated above, the embodiments described herein may include the use of a special purpose or general purpose computer (e.g., Figure 3 In addition, as indicated above, the embodiments disclosed herein may use computer-readable media (e.g., Figure 3 memory 304 or file system 306) to implement.

[0086] In some embodiments, the various components and modules described herein may be implemented as objects or processes (e.g., as separate threads) running on a computing system. Although some of the methods described herein are generally described as being implemented in software (stored on and / or run by general-purpose hardware), specific hardware implementations or combinations of software and specific hardware implementations are also possible and contemplated.

[0087] As a rule, the different features illustrated in the drawings are not necessarily drawn to scale. The illustrations provided in the present disclosure do not imply actual views of any particular device (e.g., equipment, system, etc.) or method, but rather are merely illustrative representations for describing different embodiments of the present disclosure. Accordingly, for the sake of clarity, the sizes of various features may be arbitrarily enlarged or reduced. In addition, for the sake of clarity, some drawings may be simplified. Thus, these drawings may not describe all components of a specified device (e.g., equipment) or all operations of a particular method.

[0088] The terms used herein, and especially in the appended claims (e.g., the appended claim bodies), are generally treated as “open” terms (e.g., the term “including” should be interpreted as “including but not limited to,” the term “having” should be interpreted as “having at least,” the term “comprising” should be interpreted as “including but not limited to,” etc.).

[0089] Furthermore, if it is intended to introduce a specific number in a claim recitation, that intent will be expressly recited in the claim, and in the absence of such recitation, such intent is absent. For example, to aid understanding, the following appended claims may include the use of the introductory phrases "at least one" and "one or more" to introduce claim statements. However, the use of such phrases should not be construed to imply that a claim statement introduced by the indefinite article "a" or "an" limits any particular claim containing such introduced claim statement to an embodiment containing only one such statement, even where the same claim contains the introductory phrases "one or more" or "at least one" and an indefinite article such as "a" or "an" (as an example, "a" and / or "an" should be construed to mean "at least one" or "one or more"); the same is true for the use of definite articles to introduce claim statements.

[0090] In addition, it should be understood that even if a specific number of an introduced claim recitation is explicitly recited, such recitation should be interpreted to mean at least the number recited (e.g., the expression "two recitations" without other modifiers means at least two recitations, or two or more recitations). Moreover, in those instances where a similar convention to "at least one of A, B, and C, etc." is used, generally, such a construction is intended to include A alone, B alone, C alone, A and B, A and C, B and C, or A, B and C, etc. As an example, the use of the term "and / or" should be interpreted in this manner.

[0091] In addition, any deconstructed word or phrase used to present two or more alternatives, whether in the summary of the invention, the detailed description, the claims or the drawings, should be understood to contemplate the possibility of including one, any or all of the two. For example, the phrase "A or B" should be understood to include the possibility of "A" or "B" or "A and B".

[0092] In addition, the use of the terms "first", "second", "third" and the like here is not necessarily for implying a specific order or quantity of elements. Typically, the terms "first", "second", "third" and the like are used to distinguish different elements as universal identifiers. Since the terms "first", "second", "third" and the like do not represent a specific order, these terms should not be understood to represent a specific order. In addition, in the absence of showing that the terms "first", "second", "third" and the like imply a specific number of elements, these terms should not be understood to represent a specific number of elements. For example, a first component may be described as having a first side, and a second component may be described as having a second side. Using the term "second side" relative to the second component can distinguish the side of the second component from the "first side" of the first component, rather than implying that the second component has two sides.

[0093] For illustrative purposes, the above description is described with reference to specific embodiments. However, the above illustrative discussion is not intended to be exhaustive or to limit the claimed invention to the precise form disclosed. In view of the above teachings, numerous modifications and variations are possible. These embodiments are selected and described in order to explain practical applications, thereby enabling others skilled in the art to use the claimed invention and different embodiments with different modifications suitable for the specific use contemplated.

Claims

1. A method, comprising: attaching an optical code to the patient, wherein the optical code can be sensed by an optical sensor; affixing a marking pattern to the patient at a fixed position relative to the position of the optical code, the marking pattern being perceptible by a non-optical imaging modality; capturing image data of the patient using one or more non-optical imaging modalities, the image data including an inner layer of the patient, the image data further including the pattern of markers in a fixed position relative to a position of the inner layer of the patient; Using an optical sensor of an augmented reality (AR) head mounted device to sense the optical code attached to the patient and the position of the optical code in 3D space; Accessing the image data, wherein the accessing is performed by the AR head mounted device accessing a network computer storing the image data; calculating a position of the marking pattern in the 3D space based on the sensed position of the optical code in the 3D space and the fixed position of the marking pattern relative to the position of the optical code; Based on the calculated position of the marker pattern in the 3D space and the fixed position of the marker pattern relative to the position of the inner layer of the patient in the image data, registering the position of the inner layer of the patient in the 3D space by aligning the calculated position of the marker pattern in the 3D space with the position of the marker pattern in the image data, the registration being performed by the AR head mounted device; as well as The inner layer of the patient derived from the image data projected onto an actual view of the patient is displayed in real time in the AR head mounted device based on the registration.

2. The method of claim 1, wherein attaching the optical code to the patient comprises: A bandage printed with the optical code is attached to the outer layer of the patient.

3. The method of claim 2, wherein affixing the marking pattern to the patient at the fixed position relative to the position of the optical code comprises: The marking pattern is affixed to the bandage at the fixed position relative to the position of the optical code.

4. The method according to claim 3, wherein: The marking pattern is embedded into the bandage.

5. The method according to claim 3, wherein: The marking pattern is embedded in the ink used to print the optical code on the bandage; and The ink includes a material that is sensed by the non-optical imaging modality. 6 . The method of claim 1 , wherein the optical code is a linear barcode, a matrix two-dimensional (2D) barcode, a quick response (QR) code, or a combination thereof.

7. The method of claim 1, wherein the optical code is a security credential linked to the patient's medical data, whereby the optical code is used to access the patient's medical data without requiring additional security credentials.

8. The method of claim 1 , wherein affixing the optical code to the patient comprises: The optical code is printed on the patient's skin.

9. The method of claim 1 , wherein affixing the optical code to the patient comprises: The clothing printed with the optical code is placed on the patient.

10. A device, comprising: a bandage on one side; an optical code printed on the bandage, the optical code being detectable by an optical sensor; as well as A marker pattern embedded in the ink used to print the optical code on the bandage, the marker pattern having a fixed position in the bandage relative to the position of the optical code on the bandage, the marker pattern being perceptible by a non-optical imaging mode, whereby when image data of a patient to which the device is attached is captured using one or more non-optical imaging modes, the image data includes the inner layers of the patient, and the image data further includes the marker pattern in a fixed position relative to the position of the inner layers of the patient.

11. The device of claim 10, wherein the marking pattern comprises a radiopaque material that is opaque to X-rays, a magnetically visible material, or a radioactive material. 12 . The apparatus of claim 10 , wherein the optical code is a linear barcode, a matrix two-dimensional (2D) barcode, a Quick Response (QR) code, or a combination thereof.

13. The apparatus of claim 10, wherein the optical code is a security credential linked to the patient's medical data, whereby the optical code can be used to access the patient's medical data without the need for additional security credentials.

Images