Augmented Reality Display and Tagging for Medical Procedures

Abstract

To describe a technology for augmenting a medical image for use in a medical procedure.SOLUTION: A method can include the operation of receiving an image of patient anatomy captured by a visual image camera during the medical procedure. An acquired medical image associated with the patient anatomy can then be retrieved. Another operation can associate the acquired medical image to the patient anatomy. An augmentation tag associated with a location in one layer of the acquired medical image can be retrieved. A further operation can project the acquired medical image and the augmentation tag using an augmented reality headset to form a single graphical view as an overlay to the patient anatomy in either 2D, 3D or holographic form.SELECTED DRAWING: Figure 1A

Description

[Background technology]

[0001] Mixed or augmented reality is an emerging field of computing technology that combines images from the physical and virtual computing worlds into a mixed reality world. Mixed reality encompasses a wide range of technological combinations that were previously considered solely physical reality or solely virtual reality. Mixed reality blends people, places, and objects from the physical and virtual worlds into a blended environment. Mixed reality experiences are enabled through existing commercial or custom operating systems and by using compatible virtual reality (VR) or augmented reality (AR) headsets.

[0002] Augmented reality (AR) is an example of mixed reality in which a live, direct or indirect view of a physical, real-world environment is augmented or supplemented with computer-generated sensory inputs such as audio, video, graphics, and even global positioning data. As a result, the technology can enhance the viewer's current perception of reality. Traditionally, a real-world location is displayed, and the augmentation is performed in semantic context with environmental elements. With the help of advanced AR technologies (e.g., adding computer vision and object recognition), information about the real world around the user becomes interactive and digitally manipulable. Information about the environment and its objects is overlaid on the real world. This information overlaid on the real world is a virtual image, or real information. Augmented reality brings components of the digital world into the real world as perceived by the person. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] U.S. Patent Publication No. 2014 / 0275760 Summary of the Invention [Means for solving the problem]

[0004] The present technology provides systems and methods for improving surgical, invasive radiology, cardiac, or other medical procedures using mixed reality or augmented reality devices. Augmented reality devices, such as augmented reality (AR) headsets, are used to overlay acquired or virtual medical images onto a real-world scene (e.g., what a user sees in the real world) using images projected onto partially transparent lenses or screens. In the case of surgical procedures, previously acquired medical images are used as overlay images of the patient's anatomy that can be displayed. The acquired medical images can be magnetic resonance imaging (MRI), fluoroscopy, computed tomography (CT), ultrasound, nuclear medicine, computer-generated imagery (CGI), photographs, videos, or other types of acquired or synthesized medical images. An example of a synthesized medical image is an anatomical atlas overlay on the patient's anatomy to guide surgery. The acquired medical image can be projected onto the lenses of the AR headset, which is positioned to direct the acquired medical image to the appropriate portion of the patient's anatomy being treated.

[0005] Overlaying acquired medical images assists surgeons, physicians, or other medical professionals in more accurately treating or operating on a patient's anatomy. One challenge faced by physicians and other medical professionals operating on patients is ensuring the correct patient anatomy is being operated on. Operating on the wrong person, the wrong limb, or the wrong location can lead to poor outcomes. This technology provides an augmented reality tag that more accurately identifies the patient's anatomy. A patient identifier (e.g., a physical identifier) ​​attached to the patient is also photographed, and the patient identifier is used to load the associated acquired medical image and augmented reality tag for the patient's treatment. Examples include barcodes, QR codes, patient IDs, physical tags, etc. Morphometric data can also be obtained from a real-world representation of the patient's anatomy, and the morphometric data is then compared to stored morphometric data for the limb or anatomy to be treated. If the morphometric data matches, treatment proceeds; otherwise, an error is flagged based on the morphometric data. Either of these methods (e.g., patient identifier or morphometric data) can be used to identify the patient and the region of the anatomy to be treated. [Brief explanation of the drawings]

[0006] [Figure 1A] An example of the use of an augmented reality headset to augment a medical procedure with acquired medical images. [Figure 1B] 1 is a diagram illustrating a three-dimensional magnification tag for the spleen. [Figure 1C] 10 is a description of an example of using multiple extension tags. [Figure 1D] This is a description of a 2D extension tag that is automatically converted to a 3D extension tag. [Figure 1E] 1 illustrates a two-dimensional expansion tag for the kidney. [Figure 1F] 1 illustrates a three-dimensional expansion tag for the kidney. [Figure 1G] 1 illustrates a three-dimensional tumor expansion tag. [Figure 1H] 1 illustrates a three-dimensional expansion tag for tumors covered with imaged skin. [Figure 1I] 1 illustrates a three-dimensional expansion tag for anatomical structures within the colon. [Figure 1J] 1 illustrates a three-dimensional expansion tag for anatomical structures within the colon. [Figure 2A] 1 illustrates an enhanced tag in an acquired medical image used as an overlay for the appendix. [Figure 2B] 1 illustrates an enhanced tag in an acquired medical image used as an overlay for the appendix. [Figure 2C] 1 illustrates an enhanced tag in an acquired medical image used as an overlay for the appendix. [Figure 2D] 1 illustrates an enhanced tag in an acquired medical image used as an overlay for the appendix. [Figure 3A] 1 illustrates an acquired medical image overlaid on a patient's head, depicting a cross section of the patient's neck. [Figure 3B] 1 illustrates acquired medical images being overlaid on a patient's head and hand gestures being used to control an augmented reality headset. [Figure 4] 1 illustrates an example use case of an augmented reality headset using a physical patient identifier to augment a medical procedure. [Figure 5] FIG. 1 is a block diagram illustrating an example system for augmenting a view of a medical procedure using data pulled from a data storage. [Figure 6] 1 is a flowchart illustrating a method for enlarging a view of a medical procedure. [Figure 7] 1 is a flowchart illustrating a method for enlarging a view of a medical procedure. [Figure 8] FIG. 1 is a block diagram providing an illustration of a computing device employed in current technology. DETAILED DESCRIPTION OF THE INVENTION

[0007] FIG. 1A illustrates a doctor or medical professional 110 using an AR headset 112. In one example, the AR headset 112 allows the medical professional to view through a lens that projects an augmented reality image or acquired medical image onto a translucent lens. As a more specific example, an MRI image is overlaid onto the area where surgery will be performed. The overlaid image provides a composite view with the real-world scene viewed through the translucent lens or translucent optical imaging area. This aspect of the image overlaying functionality is a camera present within the augmented reality headset to create a contextual map of the space in which the patient lies, allowing for the fusion of 3D patient data, such as a hologram of images acquired prior to surgery, with the real-world patient anatomy.

[0008] To augment a medical procedure or surgery with augmented medical images, a physician or other medical professional places an AR headset 112 in the surgical field 102. A processor associated with the AR headset 112 receives an image of the patient's anatomy 116 using the AR headset's visual imaging camera. The processor then retrieves a captured medical image 114 associated with the patient's anatomy. The captured medical image may be an MRI or CT image that is provided to the AR headset's processor and memory via a wireless connection.

[0009] The acquired medical image 114 is then associated or anchored to the patient's anatomy as identified in the real-world scene or space by the AR headset 112. Anchoring involves fixing or anchoring the acquired medical image or other related virtual image (including augmented controls) to a fixed point in the viewable real world as registered by the AR headset 112. If the physician later moves his or her viewpoint, the acquired medical image 114 remains fixed in the correct spot relative to the patient's anatomy and does not move around in the physician's field of view.

[0010] Also searchable are extension tags 118 associated with the location of one layer of the acquired medical image 114 or radiology image. The extension tags are configured to fit into the three-dimensional (3D) structure of the acquired medical image 114 or radiology image to identify anatomical structures associated with the medical procedure being performed. The extension tags may be simple geometric shapes such as circles, squares, or triangles, or more complex shapes in two or three dimensions, such as the outline of the anatomical structure of interest.

[0011] FIG. 1B illustrates the use of expansion tags to pre-label acquired medical images. When a physician performs surgery on a patient's spleen 170, the physician applies an expansion tag to the patient's spleen during diagnosis of a medical condition (e.g., splenectomy or other spleen condition). The expansion tag covers a portion of the patient's anatomy that has been loosely marked by the physician. An expansion processor then detects the edges of the anatomy or uses machine pattern recognition to expand or contract the expansion tag so that it conforms to the anatomical structure. Later, when a medical procedure, diagnosis, or surgery is performed, the physician can see the expansion tag on the anatomical structure (e.g., spleen) to better ensure clear identification of the structure for treatment.

[0012] FIG. 1C illustrates that extension tags are provided for multiple anatomical layers of an acquired medical image. For example, extension tags can identify different structures in each layer of an acquired medical image and various procedures occurring in these separate layers of a patient's anatomy. Furthermore, extension tags can be associated to form tag groups 130a-e to guide a surgeon. A more detailed example is a group of diseased anatomical structures requiring surgery. These structures can be linked together in an ordered group, guiding the surgeon from one extension tag to the next. This prevents the surgeon from missing an anatomical structure requiring surgery or skipping an important procedure. Similarly, an order can be provided to address different structures in a selected order to help the surgeon memorize unusual sequences.

[0013] As another example, the dilating tags guide the surgeon through a safe path that avoids structures such as arteries 120, which are delicate and can cause problems if damaged. In this way, the surgeon can have a mapped path of the procedure before it begins. Multiple dilating tags linked together are used to show the surgical path through the patient's anatomy. For example, tags are used to show the surgeon where the surgeon plans to cut as they pass through multiple layers. The dilating tags also show the surgeon where to cut laterally and where to move with the endoscope. This allows the surgeon to more accurately plan the surgical route in advance during surgery when using the dilating tags and acquired medical images. This reduces surgical risks and improves overall patient outcomes.

[0014] In some cases, acquired medical and radiological images are coordinated with multiple extension tags to represent a single anatomical structure across them, with each tag thus identifying that anatomical structure at a different layer.

[0015] 1A , the acquired medical image 114 and the augmented tag 118 are then projected onto the lenses of the augmented reality headset 112 to form a single graphical view for the medical professional wearing the AR headset 112. As a result, the acquired medical image 114 and the augmented tag 118 appear as if the images were overlaid directly on the patient's anatomy.

[0016] In an alternative configuration, the extension tag has a geometrically shaped tag extension 122 (FIG. 1B) extending away from the extension tag. Examples of geometric shapes are flags, polygons, rectangular boxes, pyramids, or other extensions that display information to the surgeon. The information displayed in the tag extension may identify the procedure performed on the anatomy, identify the patient, identify existing patient risks, identify drug allergies, identify known complications, or other data not directly related to the patient's anatomy but related to the patient or medical procedure.

[0017] Referring now to FIG. 1D , in one further configuration of an extension tag, a physician creates one or more 2D (two-dimensional) extension tags 150a-b, each located on one of a plurality of acquired medical images. The multiple 2D (two-dimensional) extension tags are automatically combined by an application to form a 3D extension tag or 3D shape that extends through multiple layers of the acquired medical image. For example, one method is configured to form a 3D shape 160 from the 2D tags. Similarly, a 2D image or a piece of acquired medical image can be annotated with one or more 2D tags and then extended into a 3D shape using known 3D shapes such as lines, rectangles, squares, spheres, regular polygons, irregular polygons, or other shapes selectable from a menu. Marking 3D acquired medical images can be challenging due to the many layers of such images, but enabling 2D markings that can be automatically converted to 3D images makes marking acquired medical images easier for medical professionals. The system also draws a patient outline of the structures within the patient's anatomy of interest, and then the application uses edge and feature detection to identify the remaining portions of the anatomy. This automated object detection identifies the patient's anatomy to treat using the complete outline or full shape detected by the application. For example, an organ to be removed is highlighted either by the procedure, anatomical edge detection, or by a neural network the system is trained to find the organ.

[0018] The acquired medical images are radiological images acquired using MRI (magnetic resonance imaging), magnetic resonance angiography (MRA), fMRI (functional magnetic resonance imaging), mammography, CT (computed tomography), fluoroscopy, X-ray, nuclear medicine (e.g., bone scan, thyroid scan, etc.), PET (positron emission tomography), ultrasound images, or other medical imaging techniques. The acquired medical images are images created using internal photographic images of the human body, such as colonoscopy, virtual colonoscopy, or arthroscopy. Additionally, the acquired images may include fully virtual elements that are graphically rendered in virtual 3D space and added to the acquired medical images. This may include rendered tags, rendered anatomical structures (e.g., rendered bones, nerves, tendons, etc.), rendered orthopedic hardware, or other similar virtual renderings.

[0019] FIG. 1E illustrates a two-dimensional (2D) extension tag for a kidney. In this case, a medical professional creates an extension tag 172 by annotating a 2D slice or layer of an acquired medical image. The medical professional delineates the image by carefully outlining the kidney or by selecting a point within the central mass of kidney tissue and using an application program to find the kidney's borders. This 2D extension tag can be converted into a three-dimensional (3D) extension tag as described in this disclosure. In an alternative configuration, FIG. 1E illustrates a 2D cross-sectional view of a 3D extension tag.

[0020] FIG. 1F further illustrates a three-dimensional (3D) extension tag for kidney 174. The 3D extension tag is created by a medical professional by taking the 2D tag and identifying tissue in the acquired medical image that resembles the medical professional's selected kidney tissue. Once the application identifies what it believes to be a kidney, the medical professional modifies the kidney's shape to accurately identify what needs to be addressed in the medical procedure. This includes adjusting the extension tag to identify protrusions or areas not correctly captured by the application and excluding portions that the medical professional does not want included in the extension tag. Alternatively, if desired, the medical professional annotates the kidney in full 3D. The 3D extension tag is calculated to assist the medical professional in determining the size of the anatomical structure.

[0021] FIG. 1G illustrates a three-dimensional (3D) augmentation tag for a tumor 176, where the 3D augmentation tag is shown alongside an acquired medical image depicting the patient's bowel and bone structures, with the tumor underneath the bowel structure. Within the acquired medical image, the bowel structure is turned on or off to allow a medical professional to view the augmentation tag in various contexts (e.g., with or without the bowel). FIG. 1H further illustrates the 3D augmentation tag for the tumor covered by imaged skin. This view using the imaged skin provides a different context for a medical professional needing to perform a medical procedure. As previously described, these medical images with augmentation tags are used as overlays using an AR headset.

[0022] Figures 1I and 1J illustrate 3D augmented tags of anatomical structures (e.g., polyps or tumors) in the colon. Images of the colon are created using neural network machine learning methods. Polyps or potentially cancerous structures within the colon are identified using neural network machine learning techniques. The neural network is trained using numerous medical training data cases to detect or classify polyps. Alternatively, to more easily mark polyps, a medical professional can mark them with a color tag or use the application's automated assistance.

[0023] FIG. 2A illustrates an augmented tag 210 and acquired medical image 212 used as an overlay with an AR headset for a medical procedure related to the appendix. The augmented tag 210 and acquired medical image 212 are viewed from the patient's right side. The acquired medical image 212 also illustrates skeletal structures and other patient organs. In this illustration, the augmented tag is illustrated as a dark hemispherical structure (or as a light structure in some related illustrations), while other augmented tags are used that are colored (e.g., green, red, blue, etc.) or textured (e.g., special graphic textures such as fuzz, stripes, or other textures). FIG. 2B illustrates the same augmented tag 210 and acquired medical image as viewed from the patient's left side. FIG. 2C illustrates the appendix augmented tag 210 viewed from the patient's right side and an overlay of the acquired medical image, which is a partial sagittal cross-section from the acquired medical image. FIG. 3D illustrates the augmented tag 210 and an acquired medical image, which is a cross-section in a transverse plane of the acquired medical image.

[0024] 3A illustrates the use of an augmented reality headset where acquired medical images of a patient's head and brain are anchored to a real-world view of the patient's head. As can be seen in the image, a cross-section of the patient's brain can be seen overlaid on the patient's head and neck, allowing medical personnel to view the patient's anatomy prior to any invasive procedures by the medical professional.

[0025] 3B illustrates hand gestures in front of the AR headset to control it. For example, opening and closing your hands can create zoom and zoom motions. Similarly, pinching both sides of the acquired medical image and moving your hands up and down can move multiple layers of the acquired medical image.

[0026] This technology may use an augmented reality (AR) headset, a virtual reality (VR) headset, a non-head-mounted transparent overlay system, or other types of mixed reality systems that are available or will become available. To further assist in explaining this technology, an example of an AR headset will now be described. However, the use of this technology is not limited to AR headsets alone, but may use other types of augmented reality or mixed reality systems. One example of an AR headset has a thick, visor-like band that wraps around the medical professional's head. The visor is lightweight and adjustable to fit different head sizes. In some configurations, the AR headset is wirelessly networked and provides audio output. Alternatively, the medical professional wears the AR headset like corrective glasses, with the processor portion of the AR headset worn around the person's waist or connected to a separate processing station.

[0027] The AR headset includes holographic lenses, a depth camera, over-the-ear speakers, and an on-board processor with at least one GPU (graphics processing unit). The headset also includes vents to prevent the headset from overheating. The wireless connection may be Bluetooth, Wi-Fi, ZigBee, cellular, or other types of wireless connection.

[0028] AR headsets include additional supporting sensors, such as ambient light sensors and multiple environment-sensing cameras that identify the workspace, which can work in conjunction with depth-sensing cameras to create an augmented reality environment.

[0029] AR headsets also include high-resolution cameras for taking photos and recording mixed reality video. Multiple microphones are included for capturing local audio. AR headsets include a rechargeable battery system, or the AR headset receives power from an adapter like a laptop.

[0030] Some AR headsets are controlled by hand gestures, voice, or a remote controller, which are used to control objects in the AR headset. Hand gestures are also made in front of the AR headset and captured by one or more front-facing cameras. Additionally, a pointer device such as a mouse, stylus, clicker, or other pointer device is used to control the AR headset.

[0031] The depth camera is low power and has a field of view of, for example, 60 degrees in the X axis and 60 degrees in the Y axis. Other fields of view for the depth camera can also be used. There may be multiple cameras around the headrest (i.e., front and side). These cameras capture video of the physical surroundings, track the user's hands to identify gestures, and, together with the AR headset's motion sensors, help track head movements.

[0032] The ability to composite acquired medical images, virtual models, and augmented tags into the live environment means the lenses are transparent. In one configuration with two lenses, one for each eye, the lenses are made up of three layers of glass (blue, green, and red). A projection or light engine above the lenses projects light into the headset, and tiny wavy grooves in each layer of glass diffract these light particles, causing the light to bounce back and forth, enhancing the illusion of perceiving virtual objects at virtual distances. These images are displayed as what are called holograms. As the AR headset maps the room, images can be composited into the real environment through the translucent lenses.

[0033] AR headsets can anchor or "pin" virtual images or objects relative to the real environment or room. Once a virtual object or image is locked in place in the viewable or real environment, the user can move around the virtual object or image to view the virtual object without moving the object or overlay image.

[0034] 4 illustrates an augmented view of a medical procedure 402 for a medical professional 410 using an AR headset 412. A live image of the patient's anatomy and surrounding environment or room is obtained using the AR headset's live image camera. Patient markers 420 are placed on the patient and identified within the image or video of the patient's anatomy acquired by the AR headset. The patient markers 420 include patient identification information, the operative anatomy, a patient orientation marker 422, and / or an image flip prevention tag 422. The patient orientation marker 422 and the image flip prevention tag 422 may be separate or may be combined with the patient marker 420 into a single marker or tag.

[0035] A visually scannable symbol, such as a barcode attached to the patient, provides information for retrieving the patient's identity and retrieving one associated acquired medical image 414 or multiple associated acquired images. A patient orientation marker 422 is used to ensure that the acquired medical image is oriented relative to the patient's anatomy. The patient marker 420 in the radiological image and the image orientation marker 422 are matched with the acquired medical image to match the patient orientation marker 422 on the patient's anatomy with the orientation of the acquired medical image. This avoids orientation or calibration errors when registering the acquired medical image 414 with the patient's anatomy. An image flip tag 422 is used to prevent the image from being flipped or inverted when registration is performed.

[0036] In one configuration of the technique, a barcode, QR code, or special marker is used to mark the correct patient for the medical procedure, mark the patient's right side, or mark the correct limb on which to perform the medical procedure. Optically scannable codes or markers on the patient's anatomy are also compared with patient data associated with acquired medical images to see if the patient's acquired medical images and / or anatomy match the provided patient markers 420.

[0037] The acquired medical image 414 can be associated with the patient's anatomy 416 based on the patient markers 420. This means that the correct acquired medical image 414 can be retrieved based on the patient's identity and the anatomy of the surgical target, as previously specified. The acquired medical image 414 can be anchored or fixed to the patient's anatomy based in part on the patient orientation markers 422 or image discrimination prevention tags 422. Alternatively, the acquired medical image 414 can be overlaid on the patient's anatomy by identifying the topological structure of the patient's anatomy 416. Automating this process can reduce errors and significantly reduce expensive operating room time.

[0038] The location of the patient markers 420 and associated augmented tags 418 and acquired medical or radiological images can also be retrieved, allowing the augmented tags 418 to be viewed overlaid on the patient's anatomy 416. The acquired medical image 414 and augmented tags 418 can be projected onto the lenses of the augmented reality headset 412 to form a single graphical view that is virtually overlaid on the displayed patient's anatomy.

[0039] In one configuration, the endoscopic video feeds are combined into a single graphical view. Extended tags are also used to identify the size and shape of 3D structures in the endoscopic video feed and to determine an estimate of where the endoscope is located in the patient's anatomy.

[0040] 5 illustrates a system for extending the field of view for a medical procedure, such as a surgical procedure. The system includes a camera within or associated with an AR headset 542 configured to capture live images of a patient's anatomy 510. The camera provides a live video feed of the patient's anatomy viewable by the AR headset 542 for analysis. The video is analyzed to identify environmental orientation cues, such as limb location, edge detection, depth detection, or features within the captured video image. Additionally, the live video may be recorded or streamed to another location.

[0041] The augmented reality system uses a depth or distance camera to map the patient's external contours and creates a polygonal mesh that can be compared to the surface layer of the patient's 3D data acquired with imaging techniques. Thus, both "virtual" and "real" data can be compared, like a 3D fingerprint.

[0042] The enhancement processor 520 determines whether the morphological measurements taken from the live video or images match pre-measured morphological measurements associated with the patient's anatomy retrieved using patient markers 530. For example, a patient's legs, represented in a person, have specific measurements for morphological measurements that can be matched to length, width, height, shape, or other pre-measured morphological measurements.

[0043] The acquired medical image 524 associated with the patient anatomy is then retrieved as defined by the patient markers 530 and the patient anatomy 510. The acquired medical image 540 is registered with the patient anatomy 510 using the morphological measurements and the confirmed morphological measurements.

[0044] The system includes an AR headset 542 configured to project acquired medical images 522 and augmented tags 524 onto a translucent lens of the AR headset to form a single graphical view 540 that is overlaid on the patient's anatomy displayed to a user or medical professional using the AR headset, for example, by projecting a similar image into each eye, thereby creating a 3D (three-dimensional) viewing effect.

[0045] It also provides notification to the physician or medical professional that the acquired medical image matches the patient's anatomy as defined by the morphometric measurements. In this manner, it assures the physician that the correct patient and the correct body structure (such as the correct leg or arm) are being manipulated in the correctly acquired medical image 522. In addition to checking the patient's identity via the patient marker, the retrieved information or image is compared to previously taken morphometric measurements and verified using a comparison of the actual patient's anatomy as seen in the real world. If a match is confirmed, the medical procedure is more likely to be accurate or correct.

[0046] If morphometric measurements from a patient during a medical procedure do not match those associated with a previously collected patient identifier, a warning or error is then presented to the physician or medical professional. If a match error occurs, the physician decides what information needs further verification and whether to proceed with or terminate the medical procedure. This verification is particularly important when the patient is sedated and unable to communicate with the medical professional performing the procedure.

[0047] Analyzing morphometric measurements during imaging to identify medical problems or imaging a patient's anatomical structures and then correlating the morphometric data with the treatment of the problem can result in more accurate patient treatment and avoid potential errors. As a result, this technology records morphometric measurements of the patient's anatomical aspects, or aspects of the patient's anatomy that are desired to be recorded. A comparison of the morphometric measurements of the previously acquired anatomical structures and the anatomical structures currently viewed using the AR headset during the medical procedure then provides a graphical warning to the medical professional if there is a discrepancy.

[0048] For example, if a doctor is operating on a patient's skull, the doctor may need to drill a burr hole in the skull, after which the doctor may have the patient checked by the augmented reality surgical system to ensure that the shape of the patient's skull during the medical procedure matches the same shape as information from the acquired medical images taken previously during the diagnostic phase. If the skull shapes are not identical, a warning may be flashed to the doctor or medical professional. Certain metrics of the anatomical structure, such as size, shape, fat volume, facial details, and similar skull details, may be verified against stored morphometric measurement data from previous measurements of these details for the patient.

[0049] In the context of morphometric analysis, a barcode marking system can be scanned to identify the patient. For example, an ID band can be scanned, and the barcode ID information triggers the download of the patient's acquired medical images and morphometric data. The acquisition of the acquired medical images with associated morphometric data can then be verified against morphometric data calculated from live images or medical procedures captured from the patient using an AR headset. Once the patient's identity is confirmed using the morphometric measurements, an augmented tag is then placed and combined with the acquired medical images. Finally, the surgeon begins the medical procedure, as the patient's identity, acquired medical images, morphometric measurements, and similar aspects of the surgery have been verified.

[0050] As described above, a patient marker 530 is identified within the image of the patient's anatomy 510, and the patient marker 530 includes patient identifying information and pre-measured morphometric measurements 526 stored in a database. The patient marker 530 is a scannable symbol including at least one of a 1D (one-dimensional) barcode, a 2D (two-dimensional) barcode, a photographic image, a custom-generated geometric shape, or an RFID (radio frequency identification), and information from the patient marker 530 is used to retrieve patient identification, patient information 528, and retrieve acquired medical images 522. In a more specific example, the patient marker 530 has a database key or value that is used to search or query pre-measured morphometric measurements 526 stored in another database. Once the patient's identification is obtained using a virtual scan, bracelet, RFID, or morphometric measurements, the acquired medical images 522 that match the patient's identification and other patient information are then located along with an extension tag that matches the patient's identification and acquired medical images. These extension tags are loaded and directed based on the patient's status as determined by the identification tag and / or morphometry.

[0051] The system also searches for patient orientation markers 532 and augmentation tags associated with locations (e.g., Cartesian or other relative coordinates within the acquired medical or radiological images) within the acquired medical or radiological images. The images and augmentation tags of the acquired medical images are combined into a single view using the patient markers and locations. Additionally, the patient orientation markers 532 are used or matched with image orientation tags within the acquired medical or radiological images to accurately orient the acquired medical images on the patient's anatomy.

[0052] A system including an enhanced processor also operates to identify image flip tags 532. The acquired medical image and the patient's anatomy can be aligned (e.g., rotated, flipped, or inverted) using the image flip tags so that the acquired medical image is properly oriented relative to the patient's anatomy.

[0053] In another example of the system of the present invention, the headset is a virtual reality (VR) headset into which a video feed of the patient and operating room is fed and which combines acquired medical images with the video feed, in this case the VR headset is not transparent and the combined images can be used for diagnostic or other therapeutic purposes.

[0054] In one example, morphometry is used to identify procedural road markers during medical procedures such as endoscopy, arthroscopy, and laparoscopy. For example, in a colonoscopy, certain uniquely shaped anatomical structures are marked before the medical procedure. These anatomical structures are marked using augmented tags. In one specific example, five structures are marked as road markers or milestones in the medical procedure. As the actual medical procedure progresses, the system matches one of the five structures seen through the camera image, showing the surgeon where their endoscope is located within the patient's anatomy based on viewing the physical object previously recorded using morphometry. This location is displayed as an overlay on the patient's anatomy or in combination with the acquired medical image. This matching continues until all five structures are identified during the procedure.

[0055] In a colonoscopy, where many repeating structures are present, if abnormal structures are pre-tagged, a match between the video feed from the colonoscopy and the previously tagged anatomical structures would allow the system to provide the surgeon or medical professional performing the colonoscopy with an estimated location of the endoscope within the colon. This type of estimation could similarly be applied to other types of endoscopic-style procedures that may occur in the abdomen, joints, lungs, other parts of the gastrointestinal system (e.g., small intestine or stomach), reproductive organs, urinary tract, respiratory system, etc. This technology indicates the location of the endoscope or associated procedural instruments in acquired medical images (MRI, CT scan) or a virtual image based on travel distance or matched anatomical structures within the acquired medical images.

[0056] The system also offers a multi-feed display. The visual output for the medical professional can display 1) an image of the visible skin (captured by a visible light camera), 2) an image or video from the endoscope, 3) a virtually rendered image (CGI or computer-generated image), 4) enhanced tags, and / or 5) the acquired medical image (e.g., X-ray, MRI, or CT image). This allows the medical professional to view multiple images on a single screen instead of multiple screens (e.g., one for the MRI, one for the endoscopy device, one for medical charts, etc.). This also allows the medical professional to switch between different data views. "Medical professionals" as referred to herein include physicians, physician assistants, nurse practitioners, medical professionals, and various other types of medical professionals. As referred to herein, "medical procedure" includes the science or practice of diagnosing, treating, and preventing disease. Medical procedure encompasses a variety of medical procedures aimed at maintaining and / or restoring health through the prevention and treatment of human disease. Medical procedure also applies to tasks related to health science, biomedical research, and medical technology to diagnose and treat injury or disease through therapies such as medication, surgery, psychotherapy, traction, prosthetics, biologics, and ionization.

[0057] Although the techniques of the present invention are described in the context of medicine, the techniques may alternatively be applied to other fields such as technology, science, etc., where productivity is measured by some type of unit that indicates the time, effort, skill, etc., involved in completing a task.

[0058] FIG. 6 is a flow diagram illustrating a method for augmenting medical images for use in a medical procedure. The method includes, at block 610, receiving images of a patient's anatomy using a virtual image camera. The images of the patient's anatomy are collected as the patient undergoes the medical procedure. Thereafter, at block 620, retrieved acquired medical images associated with the patient's anatomy. An example of an acquired medical image is an MRI, fMRI, or CT scan.

[0059] Thereafter, in block 630, the acquired medical image is associated with the patient's anatomy. This association involves anchoring or fixing the acquired medical image to reference points in the patient's anatomy that match points in the same anatomy in the acquired medical image. Thereafter, in block 640, an augmented tag associated with the location of a layer in the acquired medical image is searched for. The augmented tag represents an anatomical structure, an incision point, or other mapping or marking for a medical procedure. In block 650, the acquired medical image and the augmented tag are projected onto the lenses of an augmented reality headset to form a single graphical view.

[0060] 7 is a flow diagram illustrating a method for enhancing a view of a medical procedure. At block 710, the method includes receiving an image of a patient's anatomy using a live imaging camera. This is an image of the patient while the medical procedure is in progress. At block 720, patient markers are identified within the image of the subject. The patient markers may include information identifying the patient, the anatomy where the procedure is being performed, a patient orientation marker, or an image flip prevention tag.

[0061] At block 730, an acquired medical image associated with the patient's anatomy is retrieved based on a portion of the patient marker. The acquired medical image may be a radiological medical image, a computer-rendered image, or another type of visual light image. At block 740, the acquired medical image may be anchored to the patient's anatomy based on a portion of the patient orientation marker.

[0062] The augmented tags associated with the patient markers and their locations within the radiology image are retrieved at block 750. At block 760, the acquired medical image and the augmented tags are projected onto the lenses of an augmented reality headset to form a single graphical view that overlays the viewed patient anatomy.

[0063] 8 illustrates a computing device 810 on which modules of the present technology execute. A high-level example of the technology is illustrated. The computing device 810 includes one or more processors 812 in communication with a memory device 820. The computing device includes a local communication interface 818 for components within the computing device. For example, the local communication interface may be a local data bus and / or associated address or control bus, as appropriate.

[0064] Memory device 820 includes modules 824 executable by processor 812 and data for modules 824. Modules 824 perform the functions described above. A data store 822 is also located within memory device 820 for storing data related to modules 824 and other applications, along with an operating system executable by processor 812.

[0065] Other applications may also be stored in memory device 820 and executed by processor 812. The components or modules described in this description may be implemented in the form of software using a high-programming level language that is compiled, interpreted, or executed using a hybrid methodology.

[0066] The computing device also has access to I / O (input / output) devices 814 that can be used by the computing device. One example of an I / O device is a display screen that can be used to display output from the computing device. Other known I / O devices may be used with the computing device as needed. The computing device may also include networking devices 816 or similar communications devices. The networking devices 816 may be wired or wireless networking devices that connect to the Internet, a LAN, a WAN, or other computing networks.

[0067] Components or modules shown as stored in memory device 820 are executed by processor 812. The term "executable" refers to program files in a format that is executed by processor 812. For example, a program in a higher-level language may be loaded into a random-access portion of memory device 820 and compiled into machine code that is executed by processor 812, or source code may be loaded by another executable program and interpreted to generate instructions in the random-access portion of memory that are executed by the processor. An executable program may be stored in any portion or component of memory device 820. For example, memory device 820 may be random-access memory (RAM), read-only memory (ROM), flash memory, solid-state drive, memory card, hard drive, optical disk, floppy disk, magnetic tape, or other memory component. Processor 812 represents multiple processors, and memory 820 represents multiple memory units operating in parallel with the processing circuitry, thereby providing parallel processing channels for processes and data within the system. Local interface 818 is used as a network to facilitate communication between multiple processors and multiple memories. Local interface 818 may use additional systems in place to coordinate communications such as load balancing, bulk data transfers, and similar systems.

[0068] Some functional units described herein are labeled as modules to further emphasize their implementation independence. For example, a module may be implemented as a hardware circuit comprising custom VLSI circuits or gate arrays, off-the-shelf semiconductors such as logic chips, transistors, or other discrete components. A module may also be implemented as a programmable hardware device such as a field programmable gate array, programmable array logic, or programmable logic device.

[0069] Modules may also be implemented in software for execution by various types of processors. An identified module of executable code may, for example, comprise one or more blocks of computer instructions, which may be organized as an object, procedure, or function. Nevertheless, the executable files of identified modules need not be physically located together, but may comprise different instructions stored in different locations that constitute a module and, when logically combined, achieve the stated purpose of the module.

[0070] In practice, a module of executable code may be a single instruction, or multiple instructions, and may be distributed across several different code segments, among different programs, and across several memory devices. Similarly, operational data is identified and illustrated herein within modules and may be embodied in any suitable form and organized within any suitable type of data structure. The operational data may be collected as a single data set or distributed in different locations, including different storage devices. Modules may be passive or active, and may comprise agents operable to perform a desired function.

[0071] The techniques described herein also relate to computer-readable storage media, including volatile and nonvolatile, removable and non-removable media, implemented in any technology for storing information, such as computer-readable media, data structures, program modules, or other data. Computer-readable storage media may include, but are not limited to, RAM, ROM, EEPROM, flash memory, or other memory technology, CD-ROM, digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic tape, magnetic disk storage, or other magnetic storage devices, or other computer storage media used to store the desired information and the techniques described.

[0072] Devices described herein also include communications connections or network devices and network connections that enable devices to communicate with other devices. A communications connection is one example of a communication medium. Communication media typically includes any information delivery media that embodies computer-readable instructions, data structures, program modules, or other data in a modulated data signal such as a carrier wave or other transport mechanism. A "modulated data signal" means a signal that has one or more characteristics set or changed in such a manner as to encode information in the signal. By way of example, and not limitation, communication media includes wired media such as a wired network or direct-wired connection, and wireless media such as acoustic, radio frequency, infrared, and other wireless media. As used herein, the term computer-readable media includes communication media.

[0073] With reference to the examples illustrated in the drawings, the present specification has used specific language to describe the same. It is nevertheless understood that no limitation of the scope of the technology is intended thereby. Changes and further modifications of the features described herein, and additional applications of the examples described herein, will occur to those skilled in the relevant art and possess this disclosure, but are considered to be within the scope of the description.

[0074] Furthermore, the described functions, structures, and characteristics may be combined in any suitable manner into one or more examples. In the foregoing description, numerous specific details, such as various example configurations, are provided to provide a thorough understanding of the described technology. However, one skilled in the relevant art will recognize that the technology can be practiced without one or more of the specific details, or with other methods, components, devices, etc. In other instances, well-known structures or operations are not shown or described in detail to avoid obscuring aspects of the technology.

[0075] Although the subject matter has been described in language specific to structural features and / or operations, it is understood that the subject matter defined in the appended claims is not limited to the specific features and operations described above. Rather, the specific features and techniques described above are disclosed as example forms of implementing the claims. Numerous modifications and alternative arrangements can be devised without departing from the spirit and scope of the described technology.

Claims

1. An image processing method for performing image processing by a processor, comprising: obtaining an acquired medical image including one or more anatomical structures of a patient; aligning the acquired medical image with a portion of the patient's anatomy that is visible through an augmented reality headset; retrieving an extension tag associated with a location of the acquired medical image and identifying at least one anatomical structure of the acquired medical image found at that location, and further providing a plurality of extension tags for a plurality of layers of the acquired medical image to indicate a surgical path through the patient's anatomical structure, and forming a tag group by ordering and associating the plurality of extension tags; projecting the acquired medical image and the augmented tag using an augmented reality headset to form a graphical view as an overlay on the patient's anatomy; An image processing method comprising:

2. The image processing method of claim 1 , wherein the extended tags are adapted to a three-dimensional (3D) structure of the acquired medical image and identify anatomical structures associated with a medical procedure to be performed.

3. moreover, creating a plurality of two-dimensional (2D) extension tags for each of a plurality of layers of the acquired medical image; The image processing method of claim 1 , further comprising: coordinating the plurality of 2D extension tags to form a three-dimensional (3D) extension tag that extends through multiple layers of the acquired medical image.

4. The search for the extension tag further comprises:

10. The image processing method of claim 1, further comprising searching a plurality of associated extension tags to represent anatomical structures of separate layers of the acquired medical image.

5. moreover, 10. The image processing method of claim 1, comprising: causing the acquired medical image to be captured using MRI, CT scan, X-ray, ultrasound, or internal photographic imaging of the human body.

6. a readable storage medium having instructions thereon that, when executed, cause a processor to enhance a medical image of a patient; obtaining an acquired medical image including one or more anatomical structures of a patient; Identifying patient markers on the patient's anatomy that are visible through an augmented reality (AR) headset, the patient markers including information identifying the patient, information identifying the patient anatomy that is the subject of the medical procedure, a patient orientation marker, or an image flip prevention tag; retrieving acquired medical images related to the patient's anatomy based on the patient markers, the acquired medical images including images acquired of one or more anatomical features of the patient's anatomy; registering the acquired medical image viewable through the augmented reality headset to the patient's anatomy based on the patient orientation marker; retrieving extension tags associated with the patient markers and associated with locations of the acquired medical images, identifying at least one anatomical structure of the acquired medical images found at the locations; and providing a plurality of extension tags for a plurality of layers of the acquired medical images to indicate a surgical path through the patient's anatomical structures, and forming a tag group by ordering and associating the plurality of extension tags; using the AR headset to project the acquired medical image and the augmented tags to form a graphical view as an overlay on the patient's anatomy; a storage medium for causing the processor to execute the

7. moreover, aligning the patient orientation marker with an image orientation marker in the acquired medical image to align the orientation of the acquired medical image with a patient orientation marker on the patient's anatomy; The storage medium according to claim 6, which causes the processor to execute the steps of:

8. moreover, using a visually scannable symbol attached to a patient to retrieve the patient's identity and to enable retrieval of acquired medical images associated with said visually scannable symbol; The storage medium according to claim 6, which causes the processor to execute the steps of:

9. moreover, using an image inversion prevention tag to prevent the acquired medical image from being inverted relative to the patient's anatomy; The storage medium according to claim 6, which causes the processor to execute the steps of:

10. moreover, combining the endoscopic video feed with the acquired medical images and the augmented tags and overlaying them on the patient's anatomy viewable by a medical professional through the lenses of the augmented reality headset; using the extended tags to identify the size and shape of 3D structures in an endoscopic video feed and estimate where the endoscope is located within the patient's anatomy; The storage medium according to claim 6, which causes the processor to execute the steps of:

11. an augmented reality headset used to view a patient's anatomy during a medical procedure; a processor in the augmented reality headset, The processor: obtaining a captured medical image including one or more anatomical structures of the patient via the augmented reality headset; obtaining morphometric measurements of the patient's anatomy based on the medical images; identifying patient markers containing information identifying the patient's anatomy using the augmented reality headset to retrieve pre-measured morphological measurements; using the patient markers identified using the augmented reality headset to retrieve pre-measured morphological measurements associated with the patient's anatomy; determining whether the morphometric measurements of the patient's anatomy match the previously determined morphometric measurements; retrieving acquired medical images associated with the patient's anatomy as defined by the patient marker and morphological measurement matches; using the augmented reality headset and the morphometric measurements to register the acquired medical image to a portion of the patient's anatomy viewable through the augmented reality headset; retrieving an extension tag associated with a location of the acquired medical image and identifying at least one anatomical structure of the acquired medical image found at the location, and further retrieving a plurality of associated extension tags to indicate a surgical path through the patient's anatomical structure; and using the augmented reality headset to project the acquired medical images and the ordered augmented tags to form a graphical view as an overlay on the patient's anatomy; system.

12. The system of claim 11 , wherein the morphological measurements are shape, width, height, depth and contour of a limb or the patient's anatomy.

13. The system of claim 11 , wherein the acquired medical image is an MRI, CT scan, X-ray, ultrasound, or internal photographic image of the human body.

14. The system of claim 11 , wherein the acquired medical image is displayed using the augmented reality headset with the acquired medical image projected onto a translucent optical imaging area.

15. moreover, matching the patient orientation marker with the image orientation tag in the acquired medical image so that the acquired medical image can be properly oriented with the patient's anatomy; The system of claim 11 , comprising:

16. 12. The system of claim 11, wherein the patient marker is a scannable symbol including at least one of a 1D (one-dimensional) barcode, a 2D (two-dimensional) barcode, a photograph, a custom-generated geometric shape, or an RFID used to retrieve the patient's identity and retrieve the acquired medical images.

17. The processor: Identifying the image flip tag; Aligning the acquired medical image with the patient's anatomy using the anti-inversion tag to prevent the acquired medical image from being inverted relative to the patient's anatomy. The system of claim 11.

Images