Enhanced imaging and display system and methods including wound identification, analysis, and treatment information or options

Abstract

An enhanced imaging and display system and methods include wound identification, analysis, and / or treatment information and / or options. The enhanced imaging and display system is configured to display a virtual avatar having biological information with information about an identified wound overlaid thereon. The information about the identified wound may include an indication of the location of the wound on the avatar, a projectile path associated with the wound through the avatar, classification of the wound, treatment options for the identified wound, an identification of possible organs or biological tissue effected by the wound, treatment processes and / or recommendations, or any combination thereof.

Description

Enhanced Imaging and Display System and Methods Including Wound Identification, Analysis, and Treatment Information or OptionsGOVERNMENT RIGHTS

[0001] This invention was made with Government support under grant number W81XWH220132 and W81XWH2230005 awarded by the Department of Defense. The Government has certain rights in the invention.PRIORITY

[0002] This application claims priority to U.S. Application No. 63 / 532,517, fded August 14, 2023, which is incorporated by reference in its entirety into this application.BACKGROUND

[0003] Injuries can occur in situations in which medical facilities, including medical imaging apparatus, are not readily available. It would be preferrable to be able to obtain internal images to diagnose and treat the patient in these situations. For example, x-rays, ultrasound, magnetic resonance imaging, or other methods would preferably be used to obtain information about the patient’s skeletal, cardiovascular, organ, or muscle systems. However, the injury may occur in a location or environment in which the patient cannot be transported to a facility with the appropriate equipment in time for treatment. In these situations, medical treatment may need to be rendered without the assistance of knowing the internal anatomical features of the patient.

[0004] The problems of rendering medical care without medical facilities and equipment is particularly prevalent in combat situations. In combat situations, the injury suffered by the patient may be more critical, in which treatment must be rendered quickly, but the opportunity to transport the injured soldier to a proper facility in generally unavailable. Combat Casualty Care (CCC) for the wounded warfighters in Multi-Domain Operation (MDO) battlefield or austere environments makes medical knowledge, skills, and efficiency of the military medical professional paramount. Prolonged Field Care (PFC) further emphasizes the necessity to ensure maximal resources and the state-of-the-art medicine are available in the battlefield setting.SUMMARY

[0005] Systems and methods are provided herein to provide three dimensional medical visualizations of internal biology to a medical professional to help them make quick diagnosis and informed decision before and during treatment of a patient.

[0006] Exemplary embodiments of the enhanced imaging and display system and method including wound identification and analysis include an augmented reality based software tool to use with an augmented reality headset to enable medics to visualize trauma in the context of human physiology and utilize decision support tools to triage and manage traumatic wounds.

[0007] Exemplary embodiments of the enhanced imaging and display system and method including wound identification and analysis permit the visualization and identification of wounds, including location and extent and wound types.

[0008] Exemplary embodiments of the enhanced imaging and display system and method permits an augmented reality capability of superimposing a graphical representation of a health anatomy onto a real -world view of an injured person. The user of the augmented reality headset may select specific anatomical systems by verbal commands.

[0009] Exemplary embodiments of the enhanced imaging and display system and method permits hemorrhage site recognition and wound classification to the augmented reality display with trauma represented as changed in the internal anatomy finite element structure, giving the user an estimated view of potential internal damage.

[0010] A method for providing wound location and analysis, comprises providing an augmented reality system comprising a display and an image capture device; receiving one or more images from the image capture device of a subject having an injury; identifying a wound location and a wound source of the injury using an image recognition module; determining a trajectory of the wound source within the subject using a trauma assessment module; determining a sequence of tissue types along the trajectory of the projectile; estimating a volume of tissue estimated to be affected by the wound source using the sequence of tissue types; displaying a virtual avatar on the augmented reality headset including biological informationwith a virtual representation of the wound location, trajectory of the wound source, and the volume of tissue affected by the wound source indicated thereon.

[0011] Identifying the wound location of the method may include using the image recognition module to adjust a histography of the red-green-blue color channels, thresholding the histography of the red-green-blue color channels, converting to a binary quantization, inverting the binary quantization, and identifying a center of prominent features.

[0012] Identifying the wound location of the method may include using the image recognition module to split a red-green-blue color channel into a red color channel, a blue color channel, and a green color channel, remove the blue color channel, stacking the red color channel and the green color channel to create a pixel array for a plurality of pixel locations, calculate a difference of the red color channel and the green color channel at each pixel location to create an intensities array, and identify intensity maxima.

[0013] A boundary may be defined by a perimeter around areas of continuous intensity maxima, each separate area of continuous intensity maxima having its own perimeter.

[0014] The method may include determining each separate area of continuous intensity maxima as an injury and determining a type associated with each injury based on the perimeter defining the injury.

[0015] Obtaining the trajectory of the wound source of the method may include receiving a projectile identification from a user of the augmented reality headset.

[0016] The method may include identifying an exit wound location of the injury using the image recognition module, and determining the trajectory of the wound source comprises estimating a linear path between the wound location and the exit wound location.

[0017] The method may include identifying an absence of an exit wound location of the injury and determining the trajectory of the wound source comprises estimating a linear path from the wound location to an end location within the subject using tissue characteristics of the sequence of tissue types.

[0018] The method may include identifying an absence of an exit wound of the injury and determine the trajectory of the projectile comprises estimating an angle of penetration of the projectile into the body.

[0019] The method may include determining a type of projectile of the wound source causing the injury using the image recognition module.

[0020] Determining the type of projectile of the method may include determining a size of an entry wound at the wound location and selecting the type of projectile from a plurality of possible projectiles based on a range of entry wound sizes associated with each type of projectile of the plurality of possible projectiles.

[0021] The plurality of possible projectiles may be determined based on an environment in which the injury occurred.

[0022] Determining the volume of tissue estimated to be affected by the wound source of the method may include estimating a first tissue volume in contact with the projectile; estimating a third tissue volume of a maximum tissue volume affected by the wound source as it passes through the subject over time, and estimating a second tissue volume affected by the projectile at a specific instance in time.

[0023] The volume of tissue may be determined using fast-running, reduced-order physical model based on nonlinear tissue retardation and bullet tumbling dynamics.

[0024] Determining the volume of tissue of the method may include determining boundary conditions at each tissue transition of the sequence of tissue types along the trajectory.

[0025] Determining the sequence of tissue of the method may include registering the virtual avatar to a physical pose of the subject, determining a location of the wound location on the virtual avatar, and using the trajectory to select the sequence of tissue from the virtual avatar, wherein the virtual avatar includes biological information of tissue types.

[0026] Displaying the virtual avatar may include displaying the virtual avatar on a lens of the augmented reality headset in a position and size registered to a field of view through the lensof the augmented reality headset so that the virtual avatar is overlaid onto the subject when viewed by a user through the augmented reality headset.

[0027] The method may include receiving a voice command from a user of the augmented reality headset and changing a display of the virtual avatar to include different biological information.

[0028] An augmented reality system for displaying a virtual avatar having biological information, may include an augmented reality headset with a lens configured to display the virtual avatar; an image capture device to receive one or more images of a subject; a computing device in communication with the augmented reality headset, the computing device comprising a processor and memory, wherein the memory comprises non-transitory machine readable instructions that are configured to perform a method when executed by the processor, the method including: receive one or more images of the subject, create a three-dimensional surface mesh of the subject, determine a pose of the subject based on the three-dimensional surface mesh; determine a wound location from the one or more images; determine a location of the wound location on the three-dimensional surface mesh of the subject; determine a wound type from a size of the wound location; determine a projectile path within the three-dimensional surface mesh of the subject; communicate the pose, the wound location, and the projectile path to the augmented reality headset, wherein the augmented reality headset is configured to receive the pose, the wound location, and the projectile path and render the virtual avatar comprising biological information and an indication of the wound location and an indication of the projectile path.

[0029] The method employed by the augmented reality system may include using a physical model to predict a projectile velocity, yawing angle, first area of influence, second area of influence, and / or a third area of influence along a shot line that intersects with multiple tissues.DRAWINGS

[0030] FIG. 1 illustrates an exemplary patient that may be encountered outside of a medical facility that has suffered injury that may benefit from the systems and methods described herein.

[0031] FIG. 2 illustrates an exemplary field of view of a user of the enhanced imaging and display system according to embodiments described herein.

[0032] FIG. 3 illustrates an exemplary enhanced imaging and display system according to embodiments described herein.

[0033] FIG. 4 illustrates an exemplary human image having a frame structure superimposed therein.

[0034] FIG. 5 illustrates an exemplary frame structure 402 with frame reference points 502 associated therewith.

[0035] FIG. 6 illustrates a portion of the body of FIG. 4 illustrating the orientation of points on the body.

[0036] FIG. 7 illustrates the corresponding result on the reference frame and frame reference points based on the orientation of the body.

[0037] FIG. 8 illustrates an exemplary avatar having biological information with the corresponding avatar frame structure and avatar frame reference points shown.

[0038] FIG. 9 illustrates an exemplary overlay of a virtual avatar having biological information overlaid on top of a physical view of a physical body.

[0039] FIG. 10, FIG. 11, FIG. 12, and FIG. 13 illustrate exemplary virtual avatars including different biological information.

[0040] FIG. 14 illustrates an exemplary block diagram of the enhanced imaging and display system and associated method according to embodiments described herein.

[0041] FIG. 1 illustrates an exemplary method for enhanced imaging and display according to embodiments described herein.

[0042] FIG. 16 illustrates an image of a portion of a patient as received by the camera of the system described herein with areas of interest identified.

[0043] FIGS. 17A and 17B illustrate a view of different projectiles through a ballistic illustrating the different trajectory and resulting trauma.

[0044] FIGS. 18 and 19 illustrate particle based physical models of a bullet penetrating a tissue block with different velocities over time. FIG. 18 illustrates a high velocity projectile and FIG. 19 illustrates a low velocity projectile.

[0045] FIG. 20A illustrates different areas of influence superimposed on top of each other with the same center axis. FIG. 20B illustrates a first and second area of influence and the associated tissue types encountered by the first area of influence. FIG. 20C illustrates the areas of influence superimposed on the avatar including biological information.

[0046] FIG. 21 illustrates the volumes associated with the first region of influence, the second region of influence, and the third region of influence.

[0047] FIG. 22 illustrates an exemplary block diagram of the enhanced imaging and display system and associated method according to embodiments described herein including the wound identification, wound analysis, and treatment information and options.

[0048] FIG. 23 illustrates an exemplary enhanced imaging and display method with wound identification, wound analysis, and / or treatment information and options using the system described herein according to embodiments described herein.

[0049] FIG. 24 illustrates an exemplary communication flow diagram of an augmented reality headset and computing device including pose estimator module, wound identification module, and trauma computation module.DESCRIPTION

[0050] The terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting of the disclosure. As used herein, the singular forms “a,” “an” and “the” are intended to include the plural forms as well, unless the context clearly indicates otherwise. These terms are merely intended to distinguish one component from another component, and the terms do not limit the nature, sequence or order of the constituent components.

[0051] It will be further understood that the terms “comprises” and / or “comprising,” when used in this specification, specify the presence of stated features, integers, steps, operations, elements, and / or components, but do not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or groups thereof. As used herein, the term “and / or” includes any and all combinations of one or more of the associated listed items. Throughout the specification, unless explicitly described to the contrary, the word “comprise” and variations such as “comprises” or “comprising” will be understood to imply the inclusion of stated elements but not the exclusion of any other elements.

[0052] In addition, the terms “unit”, “-er”, “-or”, and “module” described in the specification mean units for processing at least one function and operation, and can be implemented by hardware components or software components and combinations thereof.

[0053] In this document, when terms such as “first” and “second” are used to modify a noun, such use is simply intended to distinguish one item from another, and is not intended to require a sequential order unless specifically stated. In addition, terms of relative position such as “vertical” and “horizontal”, or “front” and “rear”, when used, are intended to be relative to each other and need not be absolute, and only refer to one possible position of the device associated with those terms depending on the device’s orientation.

[0054] An “electronic device” or a “computing device” refers to a device that includes a processor and memory. Each device may have its own processor and / or memory, or the processor and / or memory may be shared with other devices as in a virtual machine or container arrangement. The memory may contain or receive programming instructions that, when executed by the processor, cause the electronic device to perform one or more operations according to the programming instructions.

[0055] The terms “memory,” “memory device,” “computer-readable storage medium,” “data store,” “data storage facility” and the like each refer to a non-transitory device on which computer-readable data, programming instructions or both are stored. Except where specifically stated otherwise, the terms “memory,” “memory device,” “computer-readable storage medium,” “data store,” “data storage facility” and the like are intended to include single deviceembodiments, embodiments in which multiple memory devices together or collectively store a set of data or instructions, as well as individual sectors within such devices.

[0056] The terms “processor” and “processing device” refer to a hardware component of an electronic device that is configured to execute programming instructions. Except where specifically stated otherwise, the singular term “processor” or “processing device” is intended to include both single-processing device embodiments and embodiments in which multiple processing devices together or collectively perform a process.

[0057] The terms “instructions” and “programs” may be used interchangeably herein. The instructions may be stored in object code format for direct processing by the processor, or in any other computing device language, including scripts or collections of independent source code modules that are interpreted on demand or compiled in advance. Functions, methods, and routines of the instructions are explained in more detail below. The instructions may be any set of instructions to be executed directly (such as machine code) or indirectly (such as scripts) by the processor. For example, the instructions may be stored as computing device code on the computing device-readable medium.

[0058] The term “data” may be retrieved, stored or modified by processors in accordance with a set of instructions. For instance, although the claimed subject matter is not limited by any particular data structure, the data may be stored in computing device registers, in a relational database as a table having a plurality of different fields and records, XML documents or flat files. The data may also be formatted in any computing device-readable format.

[0059] The term “module” refers to a set of computer-readable programming instructions, as executed by a processor, that cause the processor to perform one or more specified function(s).

[0060] Although exemplary embodiments are described as using a plurality of modules to perform the exemplary process, it is understood that the exemplary processes may also be performed by one or plurality of units. Additionally, it is understood that the term controller / control unit refers to a hardware device that includes a memory and a processor and is specifically programmed to execute the processes described herein. The memory is configuredto store the modules, and the processor is specifically configured to execute these modules to perform one or more processes that are described further below. Although generally the modules are understood to be software routines stored as non-transitory machine-readable instructions that are configured to be executed by a processor to perform the functions, the modules may also or alternatively be performed by and / or in combination with hardware components.

[0061] Further, the control logic of the present disclosure, including, for example, the exemplary modules described herein, may be embodied as non-transitory computer readable media on a computer readable medium containing executable programming instructions executed by a processor, controller, or the like. Examples of computer readable media include, but are not limited to, ROM, RAM, compact disc (CD)-ROMs, magnetic tapes, floppy disks, flash drives, smart cards and optical data storage devices. The computer readable medium can also be distributed in network-coupled computer systems so that the computer readable media may be stored and executed in a distributed fashion such as, e.g., by a telematics server or a Controller Area Network (CAN).

[0062] The following detailed description illustrates by way of example, not by way of limitation, the principles of the invention. This description will clearly enable one skilled in the art to make and use the invention, and describes several embodiments, adaptations, variations, alternatives and uses of the invention, including what is presently believed to be the best mode of carrying out the invention. It should be understood that the drawings are diagrammatic and schematic representations of exemplary embodiments of the invention and are not limiting of the present invention nor are they necessarily drawn to scale.

[0063] Exemplary embodiments of the enhanced imaging and display system and methods described herein include an augmented reality tool that may be used with an augmented reality headset to visualize healthy human anatomy superimposed onto the physical body of a patient.

[0064] In an exemplary embodiment, the enhanced imaging and display system comprises a standalone system that may be used in a remote location without access to a network.

[0065] Exemplary embodiments of the enhanced imaging and display system and method described herein may use an augmented reality headset system that may be used in combination with software modules that allow for real-time viewing of avatars having biological information properly overlaid upon a real subject. The enhanced imaging and display system may use posetracking and real-time three-dimensional displays to provide more information and between capabilities for field medics for treating injuries in areas away from a medical facility, and particularly on the battlefield.

[0066] The enhanced imaging and display system described herein may also or alternatively include other features to add trauma recognition and wound visualization capabilities to provide valuable decision support to the wearer.

[0067] Exemplary embodiments described herein include an enhanced imaging and display system and methods. Exemplary embodiments described herein use an augmented reality system to overlay an avatar onto a physical object to display internal attributes of the object not physically visible.

[0068] The enhanced imaging and display system and methods may use a virtual avatar including the internal attributes that, through an augmented reality system is displayed and overlaid over a physical object as viewed through an augmented reality system. The virtual avatar of the physical object including the internal attributes to be aligned with and overlaid on the physical object.

[0069] The enhanced imaging and display system and methods may use object recognition and pose recognition to identify frame points on the object. The avatar to be overlaid on the physical object may also have its own reference frame points. The system may be configured to use the identified frame points of the physical object to modify a general avatar to reposition the avatar so that when displayed, the avatar is aligned with the physical object as viewed through the augmented reality system.

[0070] Although embodiments of the exemplary system shown and described herein are primarily in relation to virtually displaying internal information of a patient overlaid on a physical view of a body of the patient, the disclosure is not so limited. Instead, the system andmethods described herein may be used to overlay any virtual avatar over the body of a patient. In addition, the patient may not be limited to a human patient but may also be used with animals or other creatures in which the creature and the avatar have corresponding reference frame points to be aligned. Furthermore, although embodiments of the invention may be described and illustrated herein in terms of specific biological information, it may be used to overlay any desired information onto a potential patient that may be used to diagnose and / or treat the patient. For example, the avatar may include heat distribution of the patient that is virtually overlaid onto the patient to determine relative warmer and / or cooler areas of the body. Other information of the patient and / or biological information may be used in combination or alternatively with the biological information shown and described herein.

[0071] Exemplary embodiments of the enhanced imaging and display system and methods described herein are configured to display a virtual avatar of a patient over a physical view of the patient as seen through an augmented reality device. The virtual avatar optionally includes desirable information related to the treatment and / or diagnosis of the patient. For example, the virtual avatar may include biological information as shown and described herein.

[0072] FIG. 1 illustrates an exemplary patient that may be encountered outside of a medical facility that has suffered injury. The patient 102 as illustrated has suffered numerous injuries 104 including a missing limb and bullet wound. A field personnel 106 treating the patient 102 may use an augmented reality headset 108 according to embodiments of the enhanced imaging and display system described herein to view a virtual avatar overlaid onto the physical body 110 of the patient 102.

[0073] FIG. 2 illustrates an exemplary field of view of a user of the enhanced imaging and display system according to embodiments described herein. As illustrated, a user, such as field personnel 106, may observe a physical environment 202. The user may be wearing an augmented reality headset 108 as illustrated in FIG. 1 having a lens 202 for displaying a virtual object over the physical environment 202 as viewed through the lens. The virtual object may be an avatar 204 that includes biological information to be overlaid on the physical body 110 of the patient to be treated as seen in the field of view of the user through the lens 202 of the augmented reality headset 108.

[0074] FIG. 3 illustrates schematically an exemplary enhanced imaging and display system according to embodiments described herein. The system may include an augmented reality headset 108 and a computing unit 304.

[0075] The augmented reality headset 108 may include a harness (as seen in FIG. 1) for positioning the headset on a user’s head and a lens 206 for displaying a virtual object 206 within the field of view 302. The virtual object 206 is configured by the system to be aligned and overlaid on top of the physical object 110.

[0076] The augmented reality headset 108 may also include an image capture module 310. The image capture module 310 may include the camera 312 and / or may be in communication with the camera and receive images from the camera. The image capture module 310 may be configured to receive one or more images of the physical environment including the physical object related to the virtual avatar. Optionally, as described in more detail herein, the physical object is a patient and the virtual object 204 is a representation of biological information related to the patient to be virtually overlaid on top of the field of view of the user including the physical body 110 of the patient 102. In this case, the camera is configured and oriented to take images of the physical body as the user is looking at the physical body through the lens 206 of the augmented reality headset 108. The image capture module 310 is configured to take desired images from the camera 312.

[0077] In an exemplary embodiment, the camera 312 is attached to and supported by the augmented reality headset 108. The camera may therefore have a field of view 306 in a known relationship to the field of view 302 of the user when wearing the augmented reality headset 108 as seen through the lens 206 of the headset. The known relationship between the field of view 306 of the camera 312 and the field of view 302 of the user may be used to properly orient the virtual avatar 204 in relation to the physical body 110.

[0078] The augmented reality headset 108 may also include an image capture module 310. The image capture module 310 may include the camera 312 and / or may be in communication with the camera and receive images from the camera. The image capture module 310 may be configured to receive one or more images of the physical environment including the physical object related to the virtual avatar. Optionally, as described in more detailherein, the physical object is a patient and the virtual object 204 is a representation of biological information related to the patient to be virtually overlaid on top of the field of view of the user including the physical body 110 of the patient 102. In this case, the camera is configured and oriented to take images of the physical body as the user is looking at the physical body through the lens 206 of the augmented reality headset 108. The image capture module 310 is configured to take desired images from the camera 312.

[0079] In an exemplary embodiment, the image capture module 310 may be configured to transform the two-dimensional image of the camera into a three-dimensional representation. For example, the enhanced imaging and display system may include one or more depth sensors. In an exemplary embodiment, the image from the camera may be used by using the real-time red-green-blue (RGB) map to estimate the depth of points on the image. The image recognition and three-dimensional modeling of the returned image may be performed by the image capture module on the augmented reality headset and / or on the computing unit.

[0080] The enhanced imaging and display system may include a computing unit 304. The augmented reality headset 108 and the computing unit 304 may be digitally connected. The digital connection may permit transfer of data between the augmented reality headset and the computing unit. For example, the augmented reality headset 108 may be separate from the computing unit 304 but may be communicatively coupled with a cable and / or wirelessly. The computing unit 304 may be used to support the augmented reality headset 108 by providing the processing power needed to render the virtual avatar according to embodiments described herein and / or to reduce the weight of the headset to be worn by the user.

[0081] The computing unit 304 may include processor and memory to perform the functions described herein. The computing unit 304 may include a pose recognition module 314 for determining the position of the physical body 110.

[0082] The computing unit 304 may be configured to receive one or more images and / or model of the physical image from the augmented reality headset, such as from the image capture module 310. The pose recognition module 314 may analyze the received image(s) to detect a target object (such as the patient to be treated) and determine a position of the target object (recognize the object’s pose).

[0083] The computing unit 304 may use the determined position of the body as determined by the pose recognition module 314 to adjust the virtual avatar to be displayed on the augmented reality headset in order to align the virtual avatar for display over the physical body.

[0084] As shown and described herein, determining a position of a target object such as a patient to be treated may include determine defining a reference frame for the target object, identifying frame reference points corresponding to specific locations of the target object, and defining a frame structure based on the frame reference points.

[0085] Exemplary embodiments of the body tracking system may determine and define frame reference points. Exemplary embodiments of the avatar may have defined therewith one or more avatar frame reference points. Once the frame reference points or the frame structure of the physical body are determined, the computing unit 304 may adjust a virtual avatar to be displayed corresponding with the determined position of the target object. The adjusted virtual avatar may be configured and / or positioned with its frame reference points aligned and / or corresponding to the frame reference points as determined for the physical body. The avatar may therefore be repositioned to be aligned and overlaid on top of the physical body as viewed through the augmented reality headset.

[0086] In an exemplary embodiment, the computing unit 304 is configured to communicate the adjusted avatar to the augmented reality headset 108 such as through the avatar display module 308. The avatar display module 308 is configured to create and display a virtual object on the lens 206 of the augmented reality headset 108 to provide an augmented view of the physical environment. In this case, the view of the physical environment is augmented by the overlay of an avatar 204 including biological information overlaid onto the physical body 110 of a patient to be treated.

[0087] The augmented reality headset 108 may include a user interface 316 for receiving inputs from the user of the augmented reality headset 108 and / or in controlling one or more attributes of the system. For example, the user interface may communicate with and / or include a microphone 318 for receiving user commands. As another example that may be used in place of or in addition to the microphone 318, the camera 312 may be used for gesture recognition to receive commands for the system described herein.

[0088] The augmented reality headset 108 may include a user interface 316 for providing information to the user of the augmented reality headset 108. The user interface 316 may be configured to display one or more virtual objects through the lens 206 of the augmented reality headset 108 to provide information to the user. For example, information may include any combination of: procedural instructions for treating the identified patient, voice command options for controlling the system (such as, for example, commands for changing the selected avatars showing biological information as described herein), menu options, etc.

[0089] Exemplary embodiments described herein may use the avatar of biological information and superimpose the avatar of biological information onto the physical view of the patient to be treated using an augmented reality headset.

[0090] In an exemplary embodiment, the avatar of biological information may be repositioned and / or reconfigured to align the avatar of biological information with the physical body of the patient to be treated as viewed by a user of the augmented reality system.

[0091] In an exemplary embodiment, the avatar of biological information may be repositioned and / or deformed at run time so that the avatar of biological information may be continuously or periodically updated to align with the physical body of the patient to be treated as viewed by a user of the augmented reality system in approximate real time. Exemplary embodiments may permit the updating of the avatar so that the avatar continues to be aligned with the patient during treatment and as the patient may move or be repositioned during treatment. It is understood that real-time updating includes the update of the avatar during the procedure that is occurring to permit the avatar to align with the physical position of the patient during the procedure. It is understood that some lag time may still be included to account for the computations described herein, or that the update may occur periodically to reduce data transfer rates and conserve power and / or processing. Real-time processing still encompasses the delayed update of the avatar if the avatar is updated during a procedure in a timeframe that is timely for the procedure being performed.

[0092] There may be instances in which the physical body does not include expected anatomical parts. For example, a limb or other body part may be missing from the physical body. Body parts may also or alternatively be out of a field of view of the user and / or out of theimage captured by the enhanced imaging and display system. There may, therefore, be missing information from the frame reference for aligning the virtual avatar. In an exemplary embodiment, the enhanced imaging and display system and methods associated therewith according to embodiments described herein may approximate a location of the body part that is missing.

[0093] Exemplary embodiments of the enhanced imaging and display system and methods described herein estimate a pose of the physical body to reposition the avatar to align the virtual display of the avatar with the physical view of the body as seen through a virtual reality device.

[0094] Exemplary embodiments of the method and system described herein may use one or more sensors for body tracking. The sensors may include one or more cameras that may take still and / or video feed images.

[0095] Exemplary embodiments of the method and system described herein may include a body tracking software system such as the pose recognition modules described herein, that is configured to receive an image of the physical body of the patient. The image may be a single still image, multiple still images, or sequential images such as from a video feed. The body tracking software system may be configured to receive the image and determine a position of the physical body from the image.

[0096] Exemplary embodiments of the body tracking system may determine and define frame reference points.

[0097] FIG. 4 illustrates an exemplary human image having a frame structure superimposed therein. As illustrated, the frame structure 402 that corresponds generally with the central axis of the body and its respective body parts.

[0098] FIG. 5 illustrates an exemplary frame structure 402 with frame reference points 502 associated therewith. As illustrated, frame reference points 504 may be used to define locations and orientations of a body relative to the frame structure.

[0099] In an exemplary embodiment, a reference frame is defined relative to the physical body at a position between the feet of the user. The frame structure 402 and frame reference points 502 may therefore be defined in position and orientation within the reference frame.

[0100] FIG. 6 illustrates a portion of the body of FIG. 4 illustrating the orientation of points on the body. FIG. 7 illustrates the corresponding result on the reference frame and frame reference points based on the orientation of the body.

[0101] In an exemplary embodiment, real-time red-green-blue (RGB) and depth images made be used to define a three-dimensional form corresponding to the physical body of the patient. The pose recognition module may be configured to determine the frame structure and / or the position and / or orientation of one or more frame reference points corresponding to the physical body.

[0102] Exemplary embodiments of the avatar may have defined therewith one or more avatar frame reference points. Once the frame reference points of the physical body are determined, the system may receive an updated avatar positioned with its frame reference points aligned and / or corresponding to the frame reference points as determined for the physical body. The avatar may therefore be repositioned to be aligned and overlaid on top of the physical body as viewed through the augmented reality headset.

[0103] FIG. 8 illustrates an exemplary avatar having biological information with the corresponding avatar frame structure and avatar frame reference points shown. The avatar may be manipulated and / or repositioned and / or reoriented such that the avatar frame structure and / or avatar frame reference points are aligned with the determined frame structure and / or frame reference points of the physical body. The avatar may therefore be resized to correspond with the size of the physical body and / or be positioned to correspond with the pose of the physical body.

[0104] FIG. 9 illustrates an exemplary overlay of a virtual avatar having biological information overlaid on top of a physical view of a physical body.

[0105] The systems and methods described herein may include one or more avatars including biological information. As described herein, the biological information may includeone or more biological features, such as any combination of one or more organs, one or more muscles, and / or one or more bones. The avatar may, for example, include any combination of a cardiovascular system, skeleton, organs, muscle system, and / or any portions thereof.

[0106] In an exemplary embodiment, the avatar including biological information may be based on an amalgamation and / or average of biological information from multiple subjects. The avatar of available information may, therefore, be an approximation of the bodily position of the biological information based on one or more subjects. In an optional embodiment, the one or more subjects used to create the avatar of the biological information may be based on attributes of a potential patient that may impact the positioning of biological information within the avatar. For example, the one or more subject may be by gender so that two different avatars are created, one male and one female as the position of biological features may change between genders. Other attributes may include age ranges, nationality, gender, prior and / or pre-existing medical conditions, and / or medical procedures undergone by a subject, etc.

[0107] In an exemplary embodiment the avatar including biological information may be selected based on a gender of the patient to be treated. For example, as described herein, an input into the system may be the gender of the patient and the system may be configured to select an avatar created from one or more subject of the same gender as the potential patient.

[0108] Exemplary avatars of biological information may include biological information received from the patient. For example, a potential patient may have one or more images performed, including, for example, x-ray, magnetic resonance imaging (MRI), computerized axial tomography (CAT) scan, ultrasound, etc. These images may be used to generate an avatar having biological information according to embodiments described herein. Specific biological information of the patient may therefore be used instead of the general biological information provided by one or more subjects. In an exemplary embodiment, the user may select a specific avatar related to a patient if it is available and / or may use a general avatar including biological information.

[0109] The avatar of biological information may be configured as a surface mesh structure. The avatar may comprise a model utilizing triangles to render the avatar in three dimensions. The vertices (X, Y, Z points) and triangles create the geometry that describesrequired points of the digital surface. The surface is then fdled in based on a control look and texture between the required points of the digital surface.

[0110] Exemplary embodiments of the avatars described herein may include low-poly mesh to provide a cleaner and smoother mesh. The resulting avatars may have less resolution of the image but may improve performance with respect to display latency. Exemplary embodiments of the avatars stored within the system may have a reduced number of polygons representing the avatar surface allowing a higher level of performance while maintaining accuracy. Low-poly mesh may comprise five (5) to fifty (50) percent of an original mesh created from an original scan or image of the biological structure.

[0111] FIGS. 10-13 illustrate exemplary virtual avatars including different biological information. FIG. 10 illustrates biological information of a cardio-vascular system. FIG. 11 illustrates biological information of a skeletal system. FIG. 12 illustrates biological information of multiple internal organ systems. FIG. 13 illustrates biological information of a musculature system.

[0112] As shown and described herein, the different biological information may be used with the virtual avatar as described herein. For example, one or more of the avatars of FIGS. 10- 13 may be used alone or combined in any combination and used to overlay on the physical view of a physical body. For example, as seen in FIG. 9, the lungs of the organ system and the skeletal system are combined as the biological information displayed on the virtual avatar 904 overlaid on the physical body 910.

[0113] Exemplary embodiments of the enhanced imaging and display system and methods described herein may include machine intelligence to define the frame reference and the reference frame points according to embodiments described herein used by the pose recognition module.

[0114] Conventional image recognition systems and methods do not reliably identify a physical body for the injury use cases anticipated for the embodiments of the present enhanced imaging and display system and methods described herein. Particularly, the image recognition systems and methods of conventional systems and methods are based on similar images or bodypositions in which the body being recognized is approximately a straight-on view in which the shoulders and hips are square and aligned so the person is essentially in a standing position. When someone is injured, however, their body positions may be contorted. For severe injuries and trauma, especially in combat areas, limbs may be missing. The image recognition systems may therefore have difficulty identifying a body and determining a body position based on the recognized body.

[0115] Exemplary embodiments of the systems and methods described herein include specifically trained computer systems based on non-conventional body images and / or positions. These body images include those in which limbs are missing, limbs are broken and unnatural anatomical locations.

[0116] In an exemplary embodiment, a computer system is used to generate body images in positions that may not correspond to full anatomical, natural positions. The computer system when creating the image may also define the reference frame corresponding to the generated body image. The generated body images with corresponding reference frame may then be used to train the machine intelligence system of the enhanced imaging and display system and methods according to embodiments described herein.

[0117] Exemplary embodiments of the enhanced imaging and display and methods described herein may improve accuracy of pose recognition of real-life wounded patients by training the computer models on simulated imagery containing a variety of human poses, wounds, uniforms, terrain, scenery, lighting, and other effects that differentiate a remote injury site, such as a battlefield, from the normal situations that pose recognition is trained on.

[0118] Exemplary embodiments of the enhanced imaging and display system and methods described herein may include a user interface for providing and / or receiving information to / from a user.

[0119] In an exemplary embodiment, the system may include an augmented reality headset configured to permit a user to view a physical environment and display a virtual object that, when viewed through the headset, overlays a virtual object onto the physical environment.In an exemplary embodiment, the virtual object may include an avatar including biological information.

[0120] Optionally, the system and methods described herein may also permit the display of additional and / or alternative information to a user. For example, as described further herein, any combination of additional information may be displayed, for example: information about the patient, information about the environment, user input options, available voice commands, suggested procedures and / or treatment options, menu options, etc.

[0121] In an exemplary embodiment, the enhanced imaging and display system and methods described herein may include a user input for receiving selections from a user of the augmented reality headset.

[0122] In an exemplary embodiment, the enhanced imaging and display system and methods may include a microphone for receiving voice commands for the system. The voice commands may include one or more words defined to control one or more attributes of the system. As an optional example, voice commands may be used to select the biological information to be displayed with the virtual avatar. For example, a user may state which biological information to overlay, such as by stating the system or organ to display. The user may state display cardiovascular system to show the heart, veins, and arteries on the virtual avatar. The voice commands may include a trigger word that is unique to provide an instruction so that the display does not change or respond unless an intentional word is used that is not used in conversation used under the circumstances of using the system. The enhanced imaging and display system and methods associated therewith may monitor the audio signals for the key word and perform functions based on the instructions received. For example, the display of the augmented reality system may be changed such as by changing the biological information of the virtual avatar based on the audio user inputs.

[0123] In an exemplary embodiment, the enhanced imaging and display system and methods may include a sensor for detecting an image or object detection. The enhanced imaging and display system and methods may be configured to recognize gestures of the user that may be used to control the system.

[0124] FIG. 14 illustrates an exemplary block diagram of the enhanced imaging and display system and associated method according to embodiments described herein.

[0125] As shown and described herein, the enhanced imaging and display system may include an image capture device. This image capture device may optionally be a camera and / or video recorder. As an optional example represented in FIG. 14, the augmented reality headset includes a video camera configured to receive a video feed.

[0126] The image recognition module 1404 may receive the video feed and may analyze one or more still images from the video feed. For example, the image recognition module may perform any combination of the following functions: extract one or more still images from the video feed, identify a surface structure from the one or more still images, and / or transform the two-dimensional image to a three-dimensional model, tessellate the surface into a multi- polygonal surface approximation.

[0127] As shown and described herein, the enhanced imaging and display system may identify and define a finite element surface mesh 1408 corresponding to the detected surface from the image recognized at step 1404 from the video feed of step 1402. The enhanced imaging and display system may, for example, tessellate a surface identified from the image recognition 1404 and generate a three-dimensional model of the surface comprising vertices corresponding to triangles (or other polygon) on the detected surface. The surface of each triangle (or other polygon) of the plurality of triangles (or other polygon) created from the tessellation of the surface approximates the surface detected from the image recognition 1404. The Image recognition module may perform the finite element surface mesh model, or the finite element surface mesh model may be handled in its own module. In an optional embodiment a finite element surface mesh module may tessellate a detected surface from an analyzed image into a multi-polygonal surface approximation.

[0128] The analyzed images, whether the three-dimensional surface model, finite element surface mesh, extracted one or more still images, and / or other result from the analysis of the video feed may be communicated to the pose recognition module 1406 and / or to the augmented reality headset to permit the overlay of the avatar thereon.

[0129] The pose recognition module 1406 may determine the pose of the target object, such as the patient to be treated. The pose recognition module 1406 may perform any combination of functions, including, without limitation: determine the presence of a target object, such as a patient to be treated; determine one or more reference frame points for defining a pose of the target object; determine one or more frame reference for defining a pose of the target object; predict and add missing information from the pose configuration, such as, for example, missing body parts of the patient to be treated. In an exemplary embodiment, the pose recognition module 1406 defines a frame reference for altering the avatar according to embodiments described herein. As shown and described herein, one or more of the functions of the pose recognition module 1406 may be performed by the image recognition module 1404 and / or the finite element surface mesh module 1408.

[0130] The enhanced imaging and display system and method described herein may use the pose recognition to adjust the avatar including biological information. For example, a skeletal avatar and an anatomy avatar including one or more organs may be adjusted to align with the pose determined by the pose recognition module 1406. For example, the avatar may include avatar frame reference and / or avatar reference frame points. These avatar frame reference and avatar reference frame points may be adjusted to correspond to the determined frame reference and / or avatar reference frame points, respectively, so that the avatar approximates the pose of the physical body.

[0131] In an exemplary embodiment, the avatar is registered and overlayed on the finite element surface mesh corresponding to the body surface of the physical body. As illustrated, the avatar skeleton is registered 1412 and overlaid on the finite element surface mesh 1408 as well as the avatar anatomy of one or more organs is registered 1410 and overlaid on the finite element surface mesh 1408 corresponding to the physical surface detected from the image of the physical body.

[0132] Any combination of avatars may be registered to model of the physical body. As illustrated, examples are provided for a skeletal avatar and an anatomy avatar. However, the system may include any number of separate and / or combined avatars. The system may be configured to select one or more avatars, such as based a programmed parameter and / or on aninput from a user. The system may then combine the avatars and / or register each avatar separately. The resulting avatar(s) are then overlaid and registered with the detected surface corresponding to the physical body. The registration may be through the overlay of the avatar to the finite element surface mesh determined to approximate the surface of the physical body as determined from image analysis of one or more images from the video feed.

[0133] The enhanced imaging and display system may thereafter display the avatar(s) including biological information through the display module 1414 of the augmented reality headset.

[0134] The enhanced imaging and display system may also be configured to update the display. As an optional example, the system may receive user inputs as described herein for altering the avatar such as by selecting or deselecting one or more avatars including biological information to add or remove from the display. As another optional example, the system may be configured to track the position and / or orientation of the headset so that the avatars remain displayed over the physical body even as the user moves their head and the physical body changes locations within the field of the of the headset. As another optional example, the system may monitor the position of the physical body and loop steps 1404, 1406, 1408, 1410, and / or 1412 to permit the avatar to be adjusted as the physical body moves within the field of the of the headset. Any combination of optional examples may be used in any combination to update the display.

[0135] In an exemplary embodiment of the enhanced imaging and display system described herein, the system may receive verbal control commands 1416. The commands may include, for example, displaying and / or not displaying one or more avatars including biological information as described more fully herein. Other verbal control commends may also be given such as in displaying or removing from the display any combination of: avatar(s), command option(s), menu(s), visual enhancements or changes (such as, for example, tone, brightness, opacity, hue, resolution, color calibration, scale, precision, alignment, etc.)

[0136] The enhanced imaging and display system may adjust the display based on an input from the user, such as through voice command detected from a microphone, gesture recognition detected from the image receiver and / or image recognition module, user console(including without limitation any combination of keyboard, buttons joystick, roller ball, touch panel, switches, etc.).

[0137] The enhanced imaging and display system may be configured to monitor, track, and / or update the virtual display based on changes in the position of the headset and / or the position of object(s) within the field of view of the headset. As an optional example, the enhanced imaging and display system may track the position of a target object, such as the patient to be treated, within the field of view of the headset. The enhanced imaging and display system may be configured to update the virtual objects (such as one or more avatars) to correspond with and / or in response to the changes in the position of the target object as determined by the tracking module.

[0138] FIG. 15 illustrates an exemplary method for enhanced imaging and display according to embodiments described herein.

[0139] As illustrated, the user may start at step 1502 by putting on an augmented reality headset according to embodiments described herein for enhanced imaging and display.

[0140] At step 1504, the user may view the patient to be treated through the augmented reality headset so that the patient. At this step the patient should be within the field of view of an image receiver, such as a video recorded and / or camera and / or other surface detector.

[0141] At step 1506, the enhanced imaging and display system optionally receive a voice or other input command from the user. In an optional embodiment, the system may receive input about taking an image of the patient and / or in confirming that the patient to be treated is in the field of view to be analyzed according to embodiments described herein. Alternate and / or additional commands and / or user inputs may be received at the enhanced imaging and display system. For example, a user may select which avatars including biological information are desired to be displayed. In an exemplary embodiment, the enhanced imaging and display system may include a plurality of different avatars comprising different biological systems, organs, etc. that a user may overlay over a field of view of the physical environment. In an optional embodiment, the user may provide information related to the patient to be used in the selection of one or more avatars according to embodiments described herein. For example, the user mayselect a gender, age, nationality, or any combination thereof that may distinguish one avatar for selection from one or more other avatars available for display. In an optional embodiment, the system may not receive voice commands but may be pre-programmed with instructions to perform pre-selected functions, including, for example, the selection of the one or more avatars. Any combination of options shown and described herein may be used in any combination.

[0142] At step 1508, the enhanced imaging and display system may receive one or more images from a surface detector as described herein. In an optional embodiment, the surface detector is a camera and / or video camera. One or more images may be received and / or extract.

[0143] At step 1510, the one or more images may be sent from the augmented reality headset to a server in communication with, but separate from, the headset. The one or more images may be analyzed by the server, such as the compute unit shown and described herein.

[0144] At step 1512, the image may be analyzed to recognize desired features within the field of view of the headset. In an exemplary embodiment, the image may be analyzed to determine a surface structure corresponding to the physical body of the patient to be treated as detected in the one or more images.

[0145] In an exemplary embodiment, the enhanced imaging and display system may tessellate the surface of the surface structure such that the tessellated surface structure comprises finite element surface mesh of multiple polygons. The finite element surface mesh is configured to approximate the physical surface of the patient detected in the one or more images.

[0146] At step 1514, the enhanced imaging and display may determine a position of the physical body of the patient captured in the one or more images. The enhanced imaging and display system may optional use image recognition to determine the position, may use the finite element surface mesh to determine the position, may determine a frame reference for the surface, may determine one or more reference frame points, and / or any combination thereof.

[0147] At step 1516, the enhanced imaging and display may register the avatar to be aligned with the physical body of the patient as detected by the system. The system may use any combination of processes described herein to reposition the avatar to correspond to the size and pose of the physical body as detected by the system. As an optional example, the avatar mayinclude reference frame points and / or a frame reference. The image analysis may determine reference frame points and / or a frame reference corresponding to the detected body as captured in the image. The avatar may be modified so that the reference frame points and / or the frame reference of the avatar match or approximates the reference frame points and / or the frame reference, respectively, of the physical body as determined by the system as captured in the image. As an optional example, the avatar may be overlaid and / or aligned with the finite element surface mesh.

[0148] At step 1518, the augmented reality display headset displays a virtual object using the lens of the headset to overlay the virtual object onto the field of view of the physical environment as seen through the headset lens. In an exemplary embodiment, the virtual object is an avatar having biological information approximating the size and / or pose of the physical body as viewed through the lens of the headset by a wearer of the headset.

[0149] At step 1520, the enhanced imaging and display system may receive one or more user inputs to control the display of the enhanced imaging and display system. In an optional embodiment, the user inputs may be received as voice commands. In an optional embodiment, the one or more user inputs may be related to the display of the augmented reality headset and / or the virtual object and / or avatar. In an optional embodiment, the augmented reality system may alter the display by repeating step 1518 based on the user input as received at step 1520.

[0150] Exemplary embodiments may use body tracking to update the position of the detected physical body and make changes to the avatar according to embodiments described herein. Any body-tracking system that provides pose estimations may be used.

[0151] In an exemplary embodiment, the pose tracking system may include a trained artificial intelligence system for determining the position of a figure. In an optional embodiment, the pose tracking system may be trained on an image set including images of bodies that do not include all anatomy and / or have anatomy in positions outside of the fully attached and correct anatomical locations. For example, images including bodies with missing body parts and / or with body parts that are broken and / or misaligned from a relaxed standing position may be used. Images including bodies with head positions not straight forward, such as bent down or to the side of the body may also or alternatively be used.

[0152] In an exemplary embodiment, high-fidelity body pose tracking using three dimensional landmarks and background segmentation masks on the whole body may be used from red-green-blue video frames.

[0153] In a preferred alternative embodiment, the enhanced imaging and display system includes a single camera or video recorded mounted on the augmented reality headset and does not require additional camera and / or sensors for image detection, body tracking, etc. Instead, exemplary embodiments of the enhanced imaging and display system described herein may use the single camera and / or video to perform multiple functions including image recognition and pose tracking.

[0154] Exemplary embodiments of the one or more avatars described herein may comprise mesh simplification to reduce the number of polygons representing a surface by fifty (50) to ninety (90) percent from an original or conventional anatomy representation. Reducing the number of polygons used to represent a surface reduce the desired framerate, such as to sixty hertz, or thirty hertz, or lower. The enhanced imaging and display system may therefore display the runtime deformed avatar that corresponded to the pose provided by the body tracking system in essentially real time.

[0155] Exemplary embodiments of the enhanced imaging and display system may run the body tracking system on a remote computing system separate from the augmented reality headset, while using the camera of the augmented reality headset without the need for multiple or separate cameras and / or sensors for position tracking. The reduction of system components is beneficial to reduce power usage (and therefore increase battery life) in the field. Other benefits may also be realized such as in the reduction of weight of the system to be used in the field. Fewer components also reduce the size and weight of supplies that need to be transported in the field.

[0156] Exemplary embodiments of the enhanced imaging and display system and methods described herein include a computer intelligence system that may be trained specifically on unique body images in which anatomy may be in unusual, such as not anatomically correct, positions and / or may be missing. The computer intelligence system may also be trained onimage sets that may cause issues with image recognition, such as in the presence of camouflage clothing, and / or different lighting conditions.

[0157] Exemplary embodiments of the enhanced imaging and display system and method shown and described herein provide an electronic trauma aid that can help field medics provide faster and better quality of care in the field without conventional and personal medical imaging equipment.

[0158] Exemplary embodiments of the enhanced imaging and display system and method described herein may provide accurate and timely placement of holographic avatar onto subjects utilizing mobile processing resources and body tracking algorithms to maintain the avatar registered to the physical environment and detected by the image processing system.

[0159] Exemplary embodiments of the enhanced imaging and display system and method described herein may include a user interface to allow field medics to command the functions described herein without creating distractions or diverting attention from treating the patient.

[0160] Exemplary embodiments of the enhanced imaging and display system and method according to embodiments described herein use an augmented reality headset, includes software and processes to receive a digital human avatar, adjust the digital human avatar to work with embodiments of the system described herein, integrate with a body tracker to support placement of a holographic projection of the digital human avatar onto a patient.

[0161] Exemplary embodiments of the enhanced imaging and display system and method described herein include an augmented reality triage tool to provide visualization of human internal anatomy from one or more anatomical avatars, superimposed on the view of an injured subject.

[0162] Exemplary embodiments of the enhanced imaging and display system and method described herein allow for the selective addition and / or removal of layers of obstructing / obscuring anatomy. For example, a user may select and / or update a selection of any combination of anatomical features to be displayed, including, for example, vasculature systems, arteries, veins, heart, organs, lungs, bones, skeleton, muscles, soft tissues, circulatory, skeletal, respiratory system, nervous system, cricothyroid, trachea, lungs, pulmonary system.

[0163] Exemplary embodiments of the enhanced imaging and display may also or alternatively permit the user to display vital signs of the patient. The system may be configured to detect a condition of the patient and / or may communicate with one or more other medical monitoring systems to receive vital signs of the patient and display the vital signs to the wearer of the augmented reality headset.

[0164] Although embodiments described herein register the avatar to be displayed as an overlay on top of the view of the physical body of a patient, other configurations may also or alternatively be used. For example, the avatar may be offset from the patient such that the avatar is displayed next to the patient so as not to obscure the actual view of the injury site. Exemplary embodiments may permit the user to provide commands to move and / or reposition the avatar relative to the physical body so that it may be directly overlaid and / or offset from the position of the physical body as seen through the augmented reality headset.

[0165] Exemplary embodiments of the system and method for enhanced imaging and display may include additional features for identifying a wound, analyze the wound, and / or provide treatment options based on the identified and / or analyzed wound.

[0166] The system and method for enhanced imaging and display including wound identification, wound analysis, and / or treatment options may determine areas of interest on an image of a patient as received from the augmented reality headset described herein that may include a wound to be analyzed according to embodiments described herein.

[0167] Exemplary embodiments of the system and method for enhanced imaging and display including wound identification, wound analysis, and / or treatment options include receiving the image as described herein, such as from the camera 312 or using the image capture module 310 or image recognition module 1404 as described herein.

[0168] Exemplary embodiments of the wound identification may be configured to use image recognition to identify areas of the image received from the headset of a patient to determine areas of interest likely to be a wound. Areas of interest likely to be a wound may optionally include physical lacerations on the surface of the body, such as from cuts, bullet wounds, blast, shrapnel, etc.

[0169] The system and method for enhanced imaging and display including wound identification may use image recognition to determine the location and / or border of a laceration on the surface of the body. The image recognition module may analyze the image received from the augmented reality headset to determine the areas of interest using differences in coloration of the skin, edge detection, and / or other image recognition methods.

[0170] In an exemplary embodiment, the image recognition module may use computer vision software that identifies any combination of the location, extent, or wound type compared to a known manual of wounds. As an optional example, the known manual of wounds may include the TC3 manual for tactical combat casualty care and wound treatment. As another optional example, the known manual of wounds may include a training set including images of various wounds and an identification of the wound boarders and / or wound identification to train an artificial intelligence identification system.

[0171] Exemplary embodiments of the image recognition module may be configured to determine any combination of: a wound type (such as a likely source of an injury, such as from a specific caliber gun, projectile, knife, blast, etc.), the severity of the wound, an entry wound location, an exit wound location, and / or an angle of entry of the projectile into the body.

[0172] In an exemplary embodiment, the image recognition module may be configured to distinguish between gunshot wounds. Optionally, the image recognition module may also determine shrapnel and / or blast wounds, extremity amputations, and other traumas.

[0173] The image recognition module may use spatial domain computations. Selected Fourier domain operations may also or alternatively be used.

[0174] The image recognition module may include processing steps that initiate by splitting the red-green-blue (RGB) color channels. The image recognition module may focus on the red channel data.

[0175] The image recognition module may adjust the histography of the data, blurring, thresholding, converting to binary quantization, inverting the data, and identify the centers of prominent features within the body limits as gunshot wounds and / or shrapnel wounds.

[0176] Exemplary embodiments of the gunshot wound processing algorithm of the image recognition module described herein may rely upon the assumption that most wounds will have fresh blood stains in the entry wound and surrounding tissue and clothing. Exemplary embodiments of the images from the AR headset may be in a 24-bit BGRA format 1280x720.

[0177] The prominence of the red blood stains may be exploited by splitting the color channels from the received image and discarding the blue channel. The red and green channels may be stacked to create a 1280x720x2 pixel array. This array can be processed to calculate the standard deviation of the intensities at each pixel location, i.e. the absolute value of the difference in the red and green images. These results are presented as a 32-bit image with intensities 0-255 greyscale. The intensities on the resulting image (the absolute value of the difference in the red and green images) are then modified (i.e. also considered Brightness / Contrast manipulation) to zero all intensities at or below approx. 50. The remaining intensity range is distributed 0-255. From the image, the intensity maxima are identified where the maxima are separated from the background by 50-70 intensity units. The location of the maxima in the image are to be identified with GSW(s).

[0178] In order to classify a wound, the area bounds for each of the wound types to be distinguished are defined. For example, an identified gun show would can be defined as having a diameter of 0.5-2 ins. Other wound types are classified by their potential diameter ranges of an entry wound.

[0179] Exemplary embodiments of the wound analysis can include image recognition as a user approaches the injured person and receive the image for analysis. The image can be analyzed to detect the wound, and determine a pixel size of each wound as seen in the received image.

[0180] Exemplary embodiments of the systems and methods described herein display a virtual representation to overlay on the wound. The system may use the geometry of the user and a distance sensor to detect the distance of the user of the augmented reality headset to the victim and the target of the images to determine a size of the virtual image to overlay onto the wound. Given the distance sensor reading, the table of actual wound size limits can be updated to give the appearance pixels at the detected distance. Optional embodiments may use pre-calculation of limits tables to save processing time. Exemplary embodiments of the classification can be completed if a wound appears and is within the limits of a predetermined wound category.

[0181] FIG. 16 illustrates an exemplary image of a patient 1602 in which wound identification areas 1604 have been identified and overlaid at a pixel level.

[0182] In an exemplary embodiment, the system and methods described herein may include a table of distances D and the maximum red area from image processing that would identify a specific wound or recognize that the red area in the image exceeds that of a classification of a type of wound, such as a gunshot wound. The system may then determine a classification of the wound based on the size of the detected wound.

[0183] Although embodiments described herein include an image recognition module for determining one or more characteristics of the wound from the image, any of the determined characteristics may be programmed, and / or may be entered by a user, and / or may be determined by the system through image recognition. For example, the system may be programmed with presumed characteristics, such as, for example, a caliber of a projectile weapon to be used and a perpendicular penetration into the body tissue. The system may then use the image detection to overwrite or modify the presumptions with determined characteristics. The system may then or also permit a user to input one or more of the characteristics to override what is either programmed and / or determined by the system. The estimates of the wound identification may therefore be improved and based on actual observations of a medic in the field.

[0184] FIG. 16 illustrates an image of a portion of a patient as received by the camera of the system described herein with areas of interest identified. As illustrated, the image was received by the system and analyzed by the system to detect the areas of interest. As illustrated, the image is a portion of the physical body 1602 and the areas of interest 1604 are identified by the system and comprise potential wound locations.

[0185] Once a wound has been identified, it may then be analyzed to determine an associated trauma and / or trajectory of the wound as it related to the internal anatomy of the patient. For example, exemplary embodiments of the system and method for enhanced imagingand display including wound analysis may include a trauma assessment module configured to take the identified type of wound and determine a likely diagnosis of the extend of injury, i.e. the trauma associated with the wound.

[0186] In an exemplary embodiment, the trauma assessment module may be configured to determine a trajectory of the injury. The trauma assessment module may, for example, use any combination of the wound identification including a determination of the source of the injury (such as a type of weapon), the size of the wound, the shape of the wound, the deformation of the skin at the edges of the wound, or other wound feature to determine a wound entry location and orientation (for example a projectile path).

[0187] Exemplary embodiments of the wound identification and / or trauma assessment modules may use a histogram equalization algorithm to normalize images selected for trauma recognition and edge detection algorithms may be used to locate regions of shrapnel or debris damage, as well as gunshot wound.

[0188] For instances in which a projectile weapon was determined to be the source of the injury, the trauma assessment module may be configured to generate a projection model. The projection model may include a body location for an entry wound and a projectile path corresponding to a determined direction of entry of the projectile.

[0189] Exemplary embodiments of the enhanced imaging and display including wound analysis may use physical models to simulate penetrating wounds or trauma. Different numerical methods such as enhancing traditional finite element model (FEM) may be used to handle large deformation and even inverted elements. Other methods including, for example, smoothed particle method, and / or material point based method may be additionally or alternatively used.

[0190] FIGS. 17A and 17B illustrate a view of different projectiles through a ballistic illustrating the different trajectory and resulting trauma. FIG. 17A illustrates an image of a projectile from a handgun through a ballistic material in which the trajectory of the projectile is essentially linear. FIG. 17B illustrates an image of a projection from a rifle through a ballistic material in which the trajectory of the projectile is initially generally linear, but then expands as the projectile separates and becomes internal shrapnel that expands in different direction.

[0191] FIGS. 18 and 19 are particle based physical models of a bullet penetrating a tissue block with different velocities. FIG. 18 illustrates a time sequence of a high velocity projectile. FIG. 19 illustrates a time sequence of a low velocity projectile. Each image of the sequence illustrates a time elapsed snapshots of a bullet penetrating the tissue and tumbling and causing the tissue to expand with cavities and damages.

[0192] Given the variety of projectile paths that may result from a projectile entering the body, exemplary embodiments of the trauma analysis module described herein may determine a region of influence based on the determined entry wound location, severity, and wound type.

[0193] FIGS. 20A-20B illustrate exemplary areas of influence. FIG. 20A illustrates different areas of influence superimposed on top of each other with the same center axis. FIG. 20B illustrates a first and second area of influence and the associated tissue types encountered by the first area of influence. FIG. 20C illustrates the areas of influence superimposed on the avatar including biological information.

[0194] Referring to FIG. 20A, exemplary embodiments of the trauma assessment module may define different regions of influence. A first region of influence 1902 may comprise a narrow trajectory path related to the actual estimated trajectory path of the projectile based on the entry wound.

[0195] A first area of influence 1902 is determined including an entry wound at a first end 1912. The trajectory of the first area of influence is essentially linear and corresponds to the path of the projectile if unimpeded as it passes through the body. The first area of influence 1902 may be determined in any combination of ways. For example, if an entry wound and exit wound are visible, the first end and the second end of the trajectory can be detected directly by the camera and image recognition module described herein. The trauma assessment module may define a trajectory between the first end and the second end as a linear path therebetween.

[0196] A first area of influence 1902 is determined including an entry wound at a first end 1912. A trajectory may then be received. The trajectory can be received in any combination of ways. As an optional example, the trajectory may be received as a user input. As described herein, a user may provide verbal commands including, for example, a trajectory of theproj ectile. A trajectory may be received in alternative and / or additional ways. For example, the image recognition module may detect the entry wound and / or the trajectory of the projectile.

[0197] In an exemplary embodiment, the first area of influence 1902 is determined as a projectile path and a volume including the tissues in estimated to be in direct contact with the projectile.

[0198] The trauma analysis module may be configured to determine a volume along the projectile path based on the projectile angle of attack. The projectile may enter the body straight on (perpendicular to the body surface) in which the projectile may travel straight through the body. The projectile will then contact and displace the tissue along the cross section of the projectile. The first area of influence would therefore have a diameter approximately equal to the diameter of the projectile. If the projectile enters the body at an angle or having other travel attributes, the projectile may rotate, spiral, tumble, etc. during the travel to contact additional tissue. In this case the first area of influence would have a diameter greater than the diameter of the projectile. The diameter of the first area of influence may decrease in diameter as the projectile stabilizes as it passes through the body.

[0199] The trauma analysis module may be configured to determine a first end at the entry wound and a trajectory of the projectile. The trauma analysis module may be configured to determine the tissue types along the trajectory. The trauma analysis module may use the avatar including biological information to determine tissue types and thicknesses associated along the trajectory. The trauma analysis module may be configured to estimate a volume associated with the tissue in contact with the projectile. The trauma analysis module may use the tissue types along the trajectory path to determine the estimated volume associated with the tissue in contact with projectile.

[0200] The trauma analysis module may determine which tissue types may be positioned within and / or along the first area of influence. The trauma analysis module may generate a tissue profile 1908 corresponding to the tissue elements affected by the first area of influence.

[0201] In an exemplary embodiment, the trauma analysis module may use an entry wound and an orientation of the projectile to determine where in the body the first area of influence is positioned in the body.

[0202] A third region of influence 1906 may include a wider diameter including more volume of tissue than that contained in the first area of influence 1902. The third region of influence 1906 includes all of the tissue influenced by the projectile. As the projectile traverses within the body, the projectile pushes and pulls the surrounding tissues and then the tissues recoil into the void left by the projectile. A volume of tissues is therefore affected by the passing projectile. The third region of influence 1906 includes an estimate of all of the tissues effected by the passing projectile along its entire trajectory from the entry wound to the exit wound or to the estimated stopping location.

[0203] The tissues effected by the projectile as it passes through the body changes over time. The second region of influence 1904 includes the volume of tissues effected by the projectile at the time the projectile exits the body and / or when the projectile stops within the body. The second region of influence 1904 is therefore within the third area of influence 1906 and represents an instant in time as opposed to the amalgamation over a period of time.

[0204] Exemplary embodiments of the trauma analysis module shown and described herein is configured to use the tissue types as retrieved from the avatar including biological information to determine one or more regions of influence. An exemplary region of influence is determined as the tissues effected by the projectile. Tissue types may be used to obtain elasticities and stiffness of the tissue to determine the volume of tissues effected by the passing projectile.

[0205] As illustrated, the second region of influence 1904 includes a trajectory path centered around the first region of influence with a wider radius along a length of the trajectory path in which the projectile may traverse, stop, and / or exit the body. A third region of influence 1906 may include a widest trajectory path centered around the first region of influence but with a radius greater than the first region of influence and the second region of influence. The third region of influence may include a volume in which the projectile may expand and / or shrapnel of the projectile may traverse within the body.

[0206] Referring to FIG. 20B, the different regions of influence are determined by the trauma analysis module. As illustrated, the different tissue volumes and / or areas of influence as described herein may be considered cavities. The illustrated permanent cavity may be the volume or cavity created by the physical contact with a wound source. The temporary cavity may be the volume of tissues that are effected by the wound source at a specific time period. Over time, the temporary cavity may change. The maximum temporary cavity is the total volume of tissues that are affected by the wound source as it passes through the tissues (i.e. as it travels through the permanent cavity).

[0207] FIG. 20C illustrates the areas of influence superimposed over an avatar 1910 corresponding to the body and / or with biological information. In an exemplary embodiment, the trauma analysis module may position the first area of influence over the avatar including biological information. The trauma analysis module may determine the tissue types from the biological information of the avatar within the volume within the first area of influence.

[0208] Referring back to FIG. 20B, the trauma analysis module may use the first area of influence 1902 and the tissue elements encountered by a projectile within the first area of influence to determine a third area of influence 1906. The first area of influence 1902 may be a volume that estimates the projectile path and / or stop location based on the tissues likely encountered by the projectile as determined using the first area of influence. The third area of influence 1906 may be a volume that estimates the tissues effected by the projectile as the projectile passes along the trajectory. The second are of influence 1904 may be the volume of tissue effected at the time the projectile either stops within the body or as the projectile exits the body.

[0209] FIG. 21 illustrates the volumes of tissue associated with the first region of influence 1902, the second region of influence 1904, and the third region of influence 1906. As illustrated, the region of influence are configured with the entry wound corresponding to the region of influence at a first end 2002.

[0210] As illustrated in FIG. 21, a volume of tissue may include the first region of influence 1902 that is a generally cylindrical shape along a generally linear center line. The first region of influence may include a radius from the center line to create a volume of the firstregion of influence. The radius of the first region of influence may start at a first radius and may taper along the length of the first region of influence from the entry wound along the linear center line to a second radius. The radius of the first region of influence may taper inwardly so that the second radius is less than the first radius.

[0211] A volume of tissue may include the second region of influence 1904 that is generally cylindrical shaped along a generally linear center line. The second region of influence may include a radius from the center line to create a volume of the second region of influence. The radius of the second region of influence may start at a third radius and may taper along the length of the second region of influence from the entry wound along the linear center line to a fourth radius. The radius of the second region of influence may taper outwardly so that the third radius is less than the fourth radius. The first radius of the first region of influence may be approximately equal to the third radius of the second region of influence.

[0212] A volume of tissue may include the second region of influence 1904 that may taper outwardly from a center line. The additional volume of the second area of influence is determined by the alterations of the projectile path of the projectile as the projectile encounters different tissue types. The model to determine the second region of influence may determine a taper rate for a given tissue type. The resulting second region of influence may therefore be defined by step wise tapered sections in which the rate of outward taper corresponds to the type of tissue the projectile traverses based on the type of tissue determined by the first area of influence.

[0213] The volume of tissue may include the third region of influence 1906 that is generally cylindrical shaped along a generally linear center line. The third region of influence may include a radius from the center line to create a volume of the third region of influence. The radius of the third region of influence may start at a fifth radius and may taper along the length of the third region of influence from the entry wound along the linear center line to a sixth radius. The radius of the third region of influence may taper outwardly so that the fifth radius is less than the sixth radius.

[0214] The volume of tissue that may include the third region of influence may be determined by the trauma analysis module using the first region of influence. The third region ofinfluence may include a volume in which the projectile and / or its shrapnel may be affected as the projectile travels.

[0215] In an exemplary embodiment, the avatar including biological information according to embodiments described herein to include an indication that represents the trauma. The avatar including biological information according to embodiments described herein may include, for example, an indication of one or more volumes of tissue including, for example, any combination of the first area of influence, the second area of influence, and / or the third area of influence.

[0216] Exemplary embodiments of the systems and methods for the enhanced imaging and display including wound analysis may include remodeling or modification of the finite element avatar mesh based on the penetrating wound cavity and deformation. Based on the finite element deformation predicted, elements or particles exceeding a deformation threshold may be flagged for removal and the finite element avatar may be modified accordingly.

[0217] Identify regions of influence may be based on a recognized trauma location, severity, and wound type. Fast running physical models of projectile retardation in tissues may be used to compute regions of influence.

[0218] Exemplary embodiments described herein may use a medical model as the finite element avatar that may be modified through physical modeling of penetration trauma based on recognized wound information, such as, for example, position, area, and / or wound type.

[0219] A projectile penetration model could be used to determine the projectile path for a potential projectile. The projectile penetration model may include a reduced order and high- fidelity physical model. For example, a high-fidelity (2mm resolution) avatar finite element (FE) mesh with over 2 million nodes and 76 tissue types may be used. A courser model (5mm resolution) is insufficient for collision detection with small bullets. A smoothed particle hydrodynamics (SPH) method could be used for solids implemented for ballistic penetration analysis. However, high -fidelity simulations with SPH takes 10 minutes or longer to complete. Such processing delays may be impractical for immediate display for a real time environment to use the information in the field in the augmented reality system described herein.

[0220] Exemplary embodiments of the systems and methods for the enhanced imaging and display including wound analysis may include fast-running, reduced-order physical model based on nonlinear tissue retardation and bullet tumbling dynamics. Exemplary embodiments of the fast-running physical model to simulate projectile penetration wound may be used to enable holographic trauma wound visualization.

[0221] Exemplary embodiments of the enhanced imaging and display system including wound analysis includes a trajectory analysis module configured to determine any combination of: an entry wound location, a trajectory of the projectile through the body, an exit wound location, a stopping position of the projectile within the body, a volume of tissue effected by the projectile, or any combination thereof.

[0222] In an exemplary embodiment, the first area of influence 1902 is determined as a projectile path and a volume including the tissues in estimated to be in direct contact with the projectile.

[0223] Exemplary embodiments of the systems and methods for the enhanced imaging and display including wound analysis may include fast-running, reduced-order physical model based on nonlinear tissue retardation and bullet tumbling dynamics. Exemplary embodiments of the fast-running physical model to simulate projectile penetration wound may be used to enable holographic trauma wound visualization.

[0224] Exemplary embodiments of the enhanced imaging and display system including wound analysis includes a trajectory analysis module using a fast physics reduced order boundary model. Exemplary embodiments of the fast physics model define a trajectory path along a trajectory path. The tissues along the trajectory path are determined and the boundaries of each tissue type along the trajectory path are determined. The trajectory path can be subdivided along its lengths to define piecewise portions corresponding to changes in tissue types along the trajectory path. For example, a trajectory path through the head may include tissue types of skin, skull, brain, skull, skin. The fast physics model may then identify the boundary locations at the transitions at each of the tissue interfaces. The fast physics model may therefore use geometric intersection at the tissue boundary and tissue material stiffness models to determine the trajectory and / or the effects on the tissues in each tissue region. The physicsmodel therefore uses fast running tissue material retardation modeling to compute projectile velocity change and tumbling as well as tissue cavities and damage along the trajectory path to reduce processing time.

[0225] Exemplary embodiments of the trajectory analysis module may be configured to determine any combination of an entry wound location, a trajectory path, an end of the trajectory path, or tissue regions along the trajectory path, locations of tissue boundaries along the trajectory.

[0226] In an exemplary embodiment, the trajectory analysis module is configured to determine the trajectory path using an entry wound location, an exit wound location, an incoming angle of entry, and / or bullet angle relative to the incoming angle of entry.

[0227] In an exemplary embodiment, the trajectory analysis module is configured to determine boundary conditions at each of the tissue transitions along the trajectory path. The boundary conditions may include characteristics such as, for example, any combination of the projectile speed, entry location into the tissue, exit location from the tissue, an incoming angle of entry into the tissue, the bullet angle relative to the incoming angle of entry into the tissue, an exiting angle of exit from the tissue, the bullet angle relative to the exiting angle of exit from the tissue.

[0228] The trajectory analysis module may assume a linear path from the entry location to the exit location within the tissue and may determine the boundary conditions based on the geometric intersection with the tissue and the tissue characteristics encountered at the boundary. Geometry may therefore be used to identify tissue surface intersections and higher order physics models may be used to determine the projectile estimates including location, speed, angle, or any combinations thereof.

[0229] Exemplary embodiments of the physical model may be used to predict a projectile velocity, yawing angle, first area of influence, second area of influence, and / or a third area of influence along an arbitrary shot line that intersects with multiple tissues.

[0230] Exemplary embodiments of the trauma assessment described herein may run each simulation for wound generation and projection can be completed within seconds, such as under 8 seconds or under 5 seconds, or about 5 seconds.

[0231] Exemplary embodiments of the systems and methods for the enhanced imaging and display including wound analysis may include wound visualization. The wound visualization may be shown as carved elements or conical shaped temporary or permanent cavities corresponding to the first area of influence, the second area of influence, and / or the third area of influence.

[0232] Exemplary embodiments of systems and methods for the enhanced imaging and display including wound analysis may use one or more information to analyze the wound. For example, as shown and described herein, the system may optionally use image processing to identify a wound and / or determine a corresponding wound severity and / or potential projectile that could have generated the wound. In an optional embodiment that may be used in combination with or instead of the image processing, the system may optionally use one or more user inputs to identify a wound and / or determine a corresponding wound severity and potential projectile. For example, the system may permit a user to provide an input identifying the projectile that caused a wound. The system may also or alternatively permit a user to identify a location of an entry wound. The system may also or alternatively permit a user to modify a location of an entry wound determined by the system. The system may also or alternatively permit a user to identify a projectile path used for the first area of influence of a wound. The system may also or alternatively permit a user to modify a projectile path used for the first area of influence of a wound as determined by the system.

[0233] Exemplary embodiments of the systems and methods for the enhanced imaging and display may include treatment information and / or options. The treatment information and / or options may include any combination of the overlay of one or more areas of influence onto the avatar, an overlay of potential tissues or internal biological elements that may be encountered within one or more areas of influence, an identity of the potential tissues or internal biological elements that may be encountered within one or more areas of influence, suggested treatmentoptions based on the potential injury to the potential tissues or internal biological elements that may be encountered by the projectile.

[0234] In an exemplary embodiment of the systems and methods for the enhanced imaging and display including treatment information and / or options may provide a virtual avatar to be overlaid on the physical body of the patient through an augmented reality headset as described herein. In an exemplary embodiment, any combination of information may be included on or with the avatar.

[0235] Exemplary embodiments of the systems and methods for the enhanced imaging and display including treatment information and / or options may include a virtual overlay of the modified avatar onto the subject for aligned wound visualization and text diagnose information for decision support.

[0236] A wearer of the augmented reality headset may see a partially transparent projection of a virtual avatar comprising three-dimensional internal anatomy with trauma wounds. Given that the virtual avatar is displayed through the augmented reality headset as partially transparent, the wearer may still see the physical scene with the patient to be treated aligned with the avatar. In an exemplary embodiment, an enhanced capability to “look through” the body is created.

[0237] Exemplary embodiments of the systems and methods for the enhanced imaging and display including treatment information and / or options offers incredible insight into the anatomy and wound of the injured patient and provide valuable diagnosis and treatment information in austere environments.

[0238] Exemplary embodiments of the systems and methods for the enhanced imaging and display including treatment information and / or options include providing a graphical visualization of the potentially wounded anatomy based on the modified avatar determined based on the first area of influence and / or the second area of influence and / or the third area of influence.

[0239] FIG. 22 illustrates an exemplary block diagram of the enhanced imaging and display system and associated method according to embodiments described herein including the wound identification, wound analysis, and treatment information and options.

[0240] As shown and described herein, the enhanced imaging and display system may include many features as described herein in relation to FIG. 14 including the video 1402, image recognition module 1404, pose estimator module 1406, finite element surface mesh 1408, one or more avatars 1410, 1412, display module 1414, and controller 1416. These elements may have any of the features as described with respect to FIG. 14 herein.

[0241] The enhanced imaging and display system and method with wound identification, wound analysis, and / or treatment information and options may include additional modules and / or may modify modules to provide for the wound analysis and display.

[0242] The enhanced imaging and display system and method with wound identification, wound analysis, and / or treatment information and options may include a wound recognition module 2204 similar to the features of the image recognition module 1404. However, the wound recognition module 2204 may also identify an entry wound on a patient and / or determine the severity of an injury and / or determine a trajectory path of a projectile and / or determine a potential projectile to create the identified wound.

[0243] The enhanced imaging and display system and method with wound identification, wound analysis, and / or treatment information and options may also include a trauma assessment module 2218. As shown and described herein, the trauma assessment module 2218 may determine any combination of a wound entry location, wound projectile trajectory, a first area of influence, a second area of influence, a third area of influence, or one or more tissues, organs, or biological elements potentially encountered by a projectile.

[0244] The enhanced imaging and display system and method with wound identification, wound analysis, and / or treatment information and options may include a treatment guidance module 2220. The treatment guidance module 2220 may be configured to determine any combination of the likely organs traumatized by the injury, treatment options for the likelyinjuries to the likely organs traumatized by the injury, or instructions corresponding to one or more procedures that may be performed to complete one or more of the treatment options.

[0245] In an exemplary embodiment, the avatar 1410, 1412 displayed on the display module 1414 may be modified based on the information from the trauma assessment and / or the treatment guidance as described herein and / or the display may be modified based on the information from the treatment guidance 2220. As an optional example, the avatar may be modified to include a representation of a first area of influence, a second area of influence, and / or a third area of influence. As another optional example, the avatar may be modified to include the organs or tissues likely encountered by a projectile as determined by the trauma assessment module. As another optional example, the display may be modified to include written information including any combination of likely organs effected by the projectile as determined by the trauma assessment module, treatment options, treatment procedure instructions, severity of injury, or criticality of the potential injury.

[0246] Exemplary embodiments of the enhanced imaging and display system and method with wound identification, wound analysis, and / or treatment information and options register an entry wound and potential projectile area of influence registered and overlayed on the finite element surface mesh corresponding to the body surface of the physical body. The overlay may also be registered with one or more avatars including biological information to create an overlay including any combination of information as described herein.

[0247] The enhanced imaging and display system may thereafter display the avatar(s) including biological information through the display module 1414 of the augmented reality headset with any combination of: the wound analysis, area(s) of interest, area(s) of influence, injury information, treatment information, and / or treatment procedure steps.

[0248] The enhanced imaging and display system may adjust the display based on an input from the user, such as through voice command detected from a microphone, gesture recognition detected from the image receiver and / or image recognition module, user console (including without limitation any combination of keyboard, buttons joystick, roller ball, touch panel, switches, etc.).

[0249] FIG. 23 illustrates an exemplary enhanced imaging and display method with wound identification, wound analysis, and / or treatment information and options using the system described herein according to embodiments described herein. The system may include any steps as described with respect to the method of FIG. 15.

[0250] At step 2302, a user of the system may wear an augmented reality headset according to embodiments described herein.

[0251] At step 2304, the user may look at a patient through the augmented reality headset to see the potential injury of the patient. In an exemplary embodiment, the wound area may be generally cleaned and / or exposed.

[0252] At step 2306, in an optional embodiment, the system may receive one or more inputs from the user. The inputs may indicate a desired combination of display options and / or may indicate that the patient and / or wound is within a field of view of the augmented reality headset.

[0253] At step 2308, the system may receive one or more images of the patient and / or wound. Optionally, the system may take the images in response to the user input. Optionally, the system may take the images periodically and / or continually. In an exemplary embodiment, the camera is on the headset so that the image(s) are taken at the headset. The images may approximate a point of view of the patient and / or wound as the wearer.

[0254] At step 2310, the system may take the images from the headset and send the images to the computing device, a remote server from the headset. The communication between the headset and the computing device may be wired and / or wireless.

[0255] At step 2312, the computing device analyze the image. The analysis of the image may include any combination of determining a surface of the physical body of the patient, identify of a wound, or location of one or more wounds.

[0256] At step 2314, the computing device may determine a pose of the body identified in the image. The pose may include the body position and / or position of the body, limbs, and / or head of the patient.

[0257] At step 2316, the computing device may identify the wound. The wound may be identified using color assessment, edge detection, etc. The identification of a wound may include any combination of a size of an entry wound, position of an entry wound, likely source of the cause of the injury.

[0258] At step 2318, the computing device may use the identification of the wound and calculate the trauma associated with the wound. The trauma calculation may include any combination of determining a location of the entry wound, determining a projectile path through a body, determining a first area of influence associated with the projectile path, determining a second area of influence associated with the projectile path, determining a third area of influence associated with the projectile path, or determining tissue types associated with a projectile path.

[0259] As described herein, the first area of influence may include a linear projectile path of a projectile entering the body, the second area of influence may include a potential volume of tissue effected by the projectile at the end of the projectile path (either within the body as the projectile stops or as the projectile exits the body), and a third area of influence that may include a potential volume of tissue effected by the projectile as the projectile passes the tissue based on the tissues encountered.

[0260] At step 2320, the system may compute a wound model. The wound model may include an avatar comprising biological information and / or wound information. Wound information may include an overlay to include with the avatar including a potential volume of a trajectory path of a projectile, information about the tissues and / or organs potentially effected by a potential trajectory path of a projectile, and / or potential treatment options for potential injury, and / or potential treatment processes.

[0261] At step 2322, the computing device may communicate with the headset to provide a virtual display overlaying the physical environment. The computing device may send the wound model and / or pose and / or avatars and / or other information necessary to display the desired virtual objects on the headset.

[0262] At step 2324, the virtual objects may be displayed on a lens of the augmented reality headset displaying an avatar including biological information overlaid on the physicalbody of the patient. Other information according to embodiments described herein may also or alternatively be displayed including, for example, any combination of wound identification, injury trajectory, wound analysis, one or more area(s) of influence, one or more organs or tissues potentially injured by the projectile, one or more suggested treatments, one or more process steps of one or more treatments.

[0263] At step 2324, the system may monitor a position and / or orientation of the headset and / or patient to update the display to maintain the overlay in a desired relationship relative to the patient. For example, as shown and described herein, the virtual overlay is registered to and overlaid directly over the physical body. However, the virtual overlay may also or alternatively be offset from the physical body so that the virtual object does not obstruct the view of the injury on the physical body.

[0264] At step 2326, the user may provide commands to the system to make changes to the virtual objects displayed. The commands may be through voice commands, but may also or alternatively include inputs through a user input interface, controls, buttons, switches, touch interface, knobs, gesture recognition, etc. The display may be changed, such as to add and / or remove one or more avatars of biological information, add and / or remove portions of the biological information, add and / or remove information about the wound, add and / or remove one or more areas of influence, add and / or remove treatment options, add and / or remove treatment processes, add and / or remove potential organs that may be injured by the projectile, and any combination thereof.

[0265] FIG. 24 illustrates an exemplary communication flow diagram of an augmented reality headset 2402 and computing device 2404 including pose estimator module 2406, wound identification module 2408, and trauma computation module 2410. As representative by the solid lines around the augmented reality headset 2402 and the computing device 2404, the component parts are separate and in electrical communication.

[0266] At step 2412 the augmented reality headset takes an image. The image and associated metadata are sent from the augmented reality headset to the computing device at step 2414. The computing device 2404, through the pose estimator module 2404 determines a pose of the physical body as detected from the image. At step 2418, the computing device 2404 maysend key points corresponding to the pose of the physical body to the augmented reality headset. At step 2426, the augmented reality headset uses the key points to adjust one or more avatars including biological information saved at the augmented reality headset so that the one or more avatars to be displayed corresponds to the size and pose of the physical body as defined by the key points.

[0267] Optionally, at step 2420, the computing device, through the wound identification module 2408 may analyze one or more images. The images may identify areas of interest corresponding to likely wound locations.

[0268] At step 2408, the system may optionally use the wound identifier 2408 to classify the wound and determine a two-dimensional position of the entrance wound relative to the image.

[0269] At step 2410, the system, through the trauma computation 2410, classifies the wound and determines a three-dimensional position of the wound and potential trajectory relative to the internal anatomy of the patient. The trauma computation module 2410 may determine a potential linear trajectory of the projectile. The trauma computation module 2410 may identify one or more tissue types along the potential trajectory of the projectile related to the internal anatomy of the patient identified relative to the potential trajectory. The trauma computation module 2410 may determine one or more volumes defining areas of influence that correspond to potential projectile trajectories based on the one or more tissue types, projectile types, or a combination thereof.

[0270] At step 2428, the system may use the wound classification and / or potential projectile trajectory relatively to the anatomy to determine potential areas injured by the projectile and / or in treatment options available for the potential injuries.

[0271] At step 2430, the system determines an updated finite element mesh determination. The updated finite element mesh may define a virtual tessellated surface approximating the surface of the physical body of the patient as determined from the image. The mesh determination may include a location of the entry wound and / or one or more volumescorresponding to different approximations of projectile trajectories (such as the first area of influence, the second area of influence, and / or the third area of influence as described herein).

[0272] At step 2432 the mesh modifications, additions, and positions relative to the anatomy are compiled to create an image to be displayed on the augmented reality headset. The headset may select one or more avatars including different biological information. The one or more avatars may be adjusted to include the mesh modifications corresponding to the location of one or more entry wounds, one or more potential trajectory volumes, one or more potential identification of organs potentially injured by the potential trajectories, one or more treatment options, one or more treatment steps, and any combination thereof.

[0273] Finally, the virtual avatar and / or display may be displayed on the augmented reality headset including any combination of one or more avatars including different biological information, the wound information, including without limitation, any combination of the wound location, the wound injury trajectory, one or more volumes of tissue including, for example, any combination of the potential volume of trajectory of projectiles and / or shrapnel, potential identification of organs affected by the potential trajectory of the projectile and / or shrapnel, potential treatment options, and / or potential treatment steps.

[0274] Exemplary embodiments of the enhanced imaging and display system and method with wound identification, wound analysis, and / or treatment information and options may be used to provide an augmented reality triage aid for medics with visualization of trauma and decision support for combat casualty care.

[0275] Exemplary embodiments of the enhanced imaging and display system and methods described herein may provide an enhanced medic and buddy care triage and decision support aid for use in the battlefield or austere environment, improving survivability during prolonged field care and in cases of mass casualties.

[0276] Exemplary embodiments of the enhanced imaging and display system and method with wound identification, wound analysis, and / or treatment information and options include an augmented reality tool to play a role in the field of three-dimensional surgery or treatment planning. Systems and methods described herein may include a navigational aid in planningmedical interventions, followed by instant aid during the surgery or treatment by displaying otherwise obscured anatomy and nearby vessels. It can aid medical professional to provide safer and more efficient care and contribute to better patient outcomes.

[0277] Exemplary embodiments of the system described herein can be based in software and / or hardware. While some specific embodiments of the invention have been shown the invention is not to be limited to these embodiments. For example, most functions performed by electronic hardware components may be duplicated by software emulation. Thus, a software program written to accomplish those same functions may emulate the functionality of the hardware components in input-output circuitry. The invention is to be understood as not limited by the specific embodiments described herein, but only by scope of the appended claims.

[0278] As used herein, the terms "about," "substantially," or "approximately" for any numerical values, ranges, shapes, distances, relative relationships, etc. indicate a suitable dimensional tolerance that allows the part or collection of components to function for its intended purpose as described herein. Numerical ranges may also be provided herein. Unless otherwise indicated, each range is intended to include the endpoints, and any quantity within the provided range. Therefore, a range of 2-4, includes 2, 3, 4, and any subdivision between 2 and 4, such as 2.1, 2.01, and 2.001. The range also encompasses any combination of ranges, such that 2-4 includes 2-3 and 3-4.

[0279] Although embodiments of this invention have been fully described with reference to the accompanying drawings, it is to be noted that various changes and modifications will become apparent to those skilled in the art. Such changes and modifications are to be understood as being included within the scope of embodiments of this invention as defined by the appended claims. Specifically, exemplary components are described herein. Any combination of these components may be used in any combination. For example, any component, feature, step or part may be integrated, separated, sub-divided, removed, duplicated, added, or used in any combination and remain within the scope of the present disclosure. Embodiments are exemplary only, and provide an illustrative combination of features, but are not limited thereto.

[0280] The features disclosed in the foregoing description, or the following claims, or the accompanying drawings, expressed in their specific forms or in terms of a means for performingthe disclosed function, or a method or process for attaining the disclosed result, as appropriate, may, separately, or in any combination of such features, be utilized for realizing the invention in diverse forms thereof.

Claims

CLAIMSThe invention claimed is:

1. A method for providing wound location and analysis, comprising: providing an augmented reality system comprising a display and an image capture device; receiving one or more images from the image capture device of a subject having an injury; identifying a wound location and a wound source of the injury using an image recognition module; determining a trajectory of the wound source within the subject using a trauma assessment module; determining a sequence of tissue types along the trajectory of the projectile; estimating a volume of tissue estimated to be affected by the wound source using the sequence of tissue types; displaying a virtual avatar on the augmented reality headset including biological information with a virtual representation of the wound location, trajectory of the wound source, and the volume of tissue affected by the wound source indicated thereon.

2. The method of claim 1, wherein the identifying the wound location comprises using the image recognition module to adjust a histography of the red-green-blue color channels, thresholding the histography of the red-green-blue color channels, converting to a binary quantization, inverting the binary quantization, and identifying a center of prominent features.

3. The method of claim 1, wherein the identifying the wound location comprises using the image recognition module to split a red-green-blue color channel into a red color channel, a blue colorchannel, and a green color channel, remove the blue color channel, stacking the red color channel and the green color channel to create a pixel array for a plurality of pixel locations, calculate a difference of the red color channel and the green color channel at each pixel location to create an intensities array, and identify intensity maxima.

4. The method of claim 3, wherein a boundary is defined by a perimeter around areas of continuous intensity maxima, each separate area of continuous intensity maxima having its own perimeter.

5. The method of claim 4, determining each separate area of continuous intensity maxima as an injury and determining a type associated with each injury based on the perimeter defining the injury.

6. The method of claim 1, wherein the obtaining the trajectory of the wound source is by receiving a projectile identification from a user of the augmented reality headset.

7. The method of claim 1, further comprising identifying an exit wound location of the injury using the image recognition module, and determining the trajectory of the wound source comprises estimating a linear path between the wound location and the exit wound location.

8. The method of claim 1, further comprising identifying an absence of an exit wound location of the injury and determining the trajectory of the wound source comprises estimating a linear path from the wound location to an end location within the subject using tissue characteristics of the sequence of tissue types.

9. The method of claim 1, further comprising identify an absence of an exit wound of the injury and determine the trajectory of the projectile comprises estimating an angle of penetration of the projectile into the body.

10. The method of claim 1, further comprising determining a type of projectile of the wound source causing the injury using the image recognition module.

11. The method of claim 10, wherein the determining the type of projectile comprises determining a size of an entry wound at the wound location and selecting the type of projectilefrom a plurality of possible projectiles based on a range of entry wound sizes associated with each type of projectile of the plurality of possible projectiles.

12. The method of claim 11, wherein the plurality of possible projectiles is determined based on an environment in which the injury occurred.

13. The method of claim 1, wherein determining the volume of tissue estimated to be affected by the wound source using the sequence of tissue types comprises estimating a first tissue volume in contact with the projectile; estimating a third tissue volume of a maximum tissue volume affected by the wound source as it passes through the subject over time, and estimating a second tissue volume affected by the projectile at a specific instance in time.

14. The method of claim 13, wherein the volume of tissue is determined using fast-running, reduced-order physical model based on nonlinear tissue retardation and bullet tumbling dynamics.

15. The method of claim 1, wherein the determining the volume of tissue comprises determining boundary conditions at each tissue transition of the sequence of tissue types along the trajectory.

16. The method of claim 1, wherein the determining the sequence of tissue comprises registering the virtual avatar to a physical pose of the subject, determining a location of the wound location on the virtual avatar, and using the trajectory to select the sequence of tissue from the virtual avatar, wherein the virtual avatar includes biological information of tissue types.

17. The method of claim 1, wherein the displaying the virtual avatar comprises displaying the virtual avatar on a lens of the augmented reality headset in a position and size registered to a field of view through the lens of the augmented reality headset so that the virtual avatar is overlaid onto the subject when viewed by a user through the augmented reality headset.

18. The method of claim 17, further comprising receiving a voice command from a user of the augmented reality headset and changing a display of the virtual avatar to include different biological information.

19. An augmented reality system for displaying a virtual avatar having biological information, comprising:an augmented reality headset with a lens configured to display the virtual avatar; an image capture device to receive one or more images of a subject; a computing device in communication with the augmented reality headset, the computing device comprising a processor and memory, wherein the memory comprises non-transitory machine readable instructions that are configured to perform a method when executed by the processor, the method including: receive one or more images of the subject, create a three-dimensional surface mesh of the subject, determine a pose of the subject based on the three-dimensional surface mesh; determine a wound location from the one or more images; determine a location of the wound location on the three-dimensional surface mesh of the subject; determine a wound type from a size of the wound location; determine a projectile path within the three-dimensional surface mesh of the subject; communicate the pose, the wound location, and the projectile path to the augmented reality headset, wherein the augmented reality headset is configured to receive the pose, the wound location, and the projectile path and render the virtual avatar comprising biological information and an indication of the wound location and an indication of the projectile path.

20. The augmented reality system of claim 19, wherein the method of the computing device further comprises using a physical model to predict a projectile velocity, yawing angle, first area of influence, second area of influence, and / or a third area of influence along a shot line that intersects with multiple tissues.

Images