Systems and methods for determining patient condition

Abstract

Systems and methods for non-contact respiration rate detection are disclosed. In some, an infrared sensor captures video images around the mouth and nose regions of a subject. The images are fed into a facial recognition module to identify and isolate these regions. An optical flow module stabilizes the images, and a frequency analysis module analyzes thermal intensity data within specific respiratory frequency ranges. Thermal intensity fluctuations are indicative of the respiration rate. In others, a video camera captures video images. A pose detection module identifies the chest area, and an optical flow algorithm tracks movement of the chest. A signal processing module is included to amplify and filter the respiratory movements, which are indicative of the respiration rate. The video images may be captured by drones.

Description

SYSTEMS AND METHODS FOR DETERMINING PATIENT CONDITIONCROSS-REFERENCE TO RELATED APPLICATIONS

[0001] This application claims priority to U.S. Provisional Patent Application No. 65 / 537,991 , filed on September 12, 2023, which is incorporated by reference in its entirety.STATEMENT REGARDING FEDERALLY SPONSORED RESEARCH OR DEVELOPMENT

[0002] This invention was made with government support under contract number HR001124C0334 awarded by the Defense Advanced Research Projects Agency. The government has certain rights in the invention.BACKGROUND

[0003] The present disclosure relates to care, triage, and assessment of injured persons, and to improved triage and delivery of life-saving intervention (LSI) in mass casualty incidents (MCls) such as battlefields, natural disasters such as hurricanes or earthquakes, other types of disasters such as high-rise building collapse, and / or so forth. This disclosure further pertains to innovative non-contact health monitoring technologies, specifically methods utilizing infrared-based thermal imaging and video-based optical analysis techniques. These methods provide a significant advantage in scenarios where traditional contact-based respiratory monitoring is impractical or poses additional risks, such as in MCls, hazardous environments, or remote medical assessments.

[0004] Mass casualty incidents (MCI) are events where the number of patients exceeds the available healthcare resources, especially in terms of personnel. Triage and delivery of medical assistance at disaster scenes or military battlefields is challenging due to factors such as the unpredictable location of such events and the potential for mass casualties with a wide range of possible injury types, including blunt trauma, gunshot wounds, burns, traumatic brain injuries, inhalation of toxic chemicals, and / or biohazardous biological agents. Existing triage protocols, such as Sort, Assess,Lifesaving Interventions, Treatment / Transport (SALT), although widely adopted, often fail to provide adequate monitoring during secondary assessment periods. This is largely because stable casualties are often left unmonitored due to the prioritization of scarce medical personnel to more immediate needs.

[0005] Trained first responders often lack the bandwidth to monitor many individuals simultaneously for subtle changes in physiological signatures that indicate the need for LSIs. The sheer volume of information from multiple data streams from each casualty can oversaturate medical personnel, making it difficult to track trends across different time scales. Alarms on monitoring instruments are employed to alert personnel once a single value moves outside an acceptable range; however, these signals often fail to account for the holistic integration of all data streams and are commonly ignored due to frequent false positives (i.e., alarm fatigue). These challenges are compounded by the chaotic nature of MCI environments, whether on battlefields, disaster areas, or during patient transport to higher levels of care, which create high-stress and disorienting conditions that impair objective decision-making.

[0006] Continuous non-contact monitoring using drones enables rapid response interventions following unexpected events and patient health deterioration. By leveraging these technologies, the DART system ensures that patients are continuously monitored during initial assessment, evacuation, transit, and upon arrival at field hospitals or medical centers, reducing the likelihood of delayed or absent LSIs even in overwhelmed healthcare settings.

[0007] The present disclosure proposes solutions to overcome these problems and others, providing a robust system for improving triage, monitoring, and life-saving intervention in mass casualty and emergency response scenarios.BRIEF SUMMARY

[0008] Non-contact monitoring methods provide a solution by utilizing technologies that allow for the continuous assessment of vital signs without the need for direct physical interaction. Specifically, infrared (IR) sensors can detect thermal changes around the mouth and nose associated with breathing, while video-based systems can analyze chest movements to monitor respiration rates. These non-contact methods can continuouslytrack multiple physiological signatures such as heart rate, respiration, oxygen saturation, or temperature across many casualties, without overwhelming medical personnel.

[0009] The integration of infrared-based and video-based non-contact monitoring technologies into drones further enhances the capabilities of the DART system. These drones can be equipped with both IR and optical sensors to capture vital signs remotely and relay this information to a control station. By using IR imaging to identify thermal variations around the mouth and nose, and video analysis to detect subtle chest movements, the DART system can provide a more comprehensive and accurate monitoring solution that operates effectively in varied environmental conditions, such as darkness, smoke, or in areas inaccessible to first responders.

[0010] In accordance with certain illustrative embodiments disclosed herein, a system for non-contact respiration rate detection is provided. The system comprises an infrared thermal imaging sensor configured to capture thermal intensity variations in the mouth and nose regions of a subject. To accurately identify these regions, the system employs a facial recognition module, which may utilize algorithms such as the MediaPipe face mesh to detect the mouth and nose with precision. The captured thermal images are processed through an optical flow module that stabilizes the imagery, ensuring that any movement or instability does not affect the accuracy of the data. A frequency analysis module is further included, which filters the thermal intensity data to focus on specific respiratory frequency ranges, typically between 0.1 Hz and 0.8Hz, to determine the respiration rate. The infrared thermal imaging sensor operates at an exemplary frame rate in the range of about 30 to about 60 frames per second, however this range is not limiting and other frame rates may be employed to better suit different applications. An appropriate frame rate and recording time allows for the capture of sufficient data to ensure reliable and accurate monitoring of respiration rates. Additionally, a normalization module may be incorporated to remove baseline drift and other non-respiratory thermal variations, further enhancing the accuracy of the system's analysis. In some embodiments, the infrared thermal imaging sensor is mounted on an unmanned aerial vehicle (UAV), enabling the system to be deployed in remote or hazardous environments, where rapid, non-invasive assessment of multiple individuals is critical.

[0011] Another embodiment disclosed herein pertains to a system for non-contact respiration rate detection using video-based technology. This system includes a video camera designed to capture high-resolution video data of the chest area of a subject. The system employs a pose detection module that identifies and dynamically adjusts a region of interest (ROI) around the chest area, using a combination of frequency analysis and visual landmark detection to track the subject's movements. The ROI permits the system to focus on the relevant chest movements indicative of respiration. The system further utilizes an optical flow algorithm, such as the Lucas-Kanade method, to track motion within the ROI across successive video frames. To enhance the detection of subtle respiratory movements, a motion amplification module may be included, which may apply Eulerian video magnification to increase the visibility of these movements within the ROI. The signal processing module filters out non-respiratory movements by applying a frequency band-pass filter, isolating signals within the typical respiratory frequency range. The video-based system may also be mounted on a UAV, extending its applicability to large-scale disaster response scenarios, where it can be rapidly deployed to assess the respiratory status of multiple casualties from a safe distance.

[0012] In accordance with yet another illustrative embodiment, a method for noncontact monitoring of respiration rates is disclosed. The method involves capturing infrared thermal images of the mouth and nose regions of a subject using an infrared sensor, followed by the identification and isolation of these regions of interest using facial recognition techniques. The thermal intensity fluctuations over time are analyzed to determine the respiration rate, with a spectral density analysis applied to identify the dominant frequency components associated with respiration. Additionally, video data of the chest area of the subject is captured, and a region of interest around the chest area is established using pose detection. The chest movements are then tracked using optical flow analysis, and the respiration rate is calculated based on the detected movements. This method may include capturing thermal images at an appropriate frame rate and over a sufficient time period to ensure a reliable respiration rate calculation, accounting for the subject’s typical breathing patterns, which can vary by age. Facial recognition is used to ensure that the infrared thermal imaging is precisely focused on the mouth and nose regions, maximizing the accuracy of the measurement. The video data captured mayinclude multiple views of the chest area to account for different body positions and orientations, and the region of interest around the chest area may be dynamically adjusted based on changes in the subject's position or the presence of obstructions. This method also allows for the simultaneous monitoring of multiple individuals within the field of view of the sensors, making it particularly useful in scenarios where rapid assessment of several casualties is required.

[0013] These and other non-limiting aspects and / or objects of the disclosure are more particularly described below.BRIEF DESCRIPTION OF THE DRAWINGS

[0014] The following is a brief description of the drawings, which are presented for the purposes of illustrating the exemplary embodiments disclosed herein and not for the purposes of limiting the same. Various features are not drawn to scale, and their dimensions may be arbitrarily increased or reduced for clarity of discussion.

[0015] FIG. 1 is an illustrative depiction of a mass casualty incident scene and various components of the DART system, in accordance with some embodiments of the present disclosure.

[0016] FIG. 2A and FIG. 2B together form a flow chart illustrating a first method for performing triage, in accordance with some embodiments of the present disclosure.

[0017] FIG. 3 is an example illustrating showing a method for detecting blood loss from an image, in accordance with some embodiments of the present disclosure.

[0018] FIG. 4 is an illustration of a processor and memory for executing instructions corresponding to some methods described herein.

[0019] FIG. 5 is a flow chart showing a method for determining blood loss, in accordance with some embodiments of the present disclosure.

[0020] FIG. 6 is a flow chart illustrating a first method for determining a respiration rate. This method uses a thermal imaging system.

[0021] FIGS. 7a-7c illustrate various steps in the flow chart of FIG. 6.

[0022] FIG. 8 is a flow chart illustrating a second method for determining a respiration rate. This method uses a video imaging system.

[0023] FIGS. 9a-9b illustrate various steps in the flow chart of FIG. 8.

[0024] FIG. 10 is an illustration showing a heart rate determined from an image of a face, in accordance with some embodiments of the present disclosure.

[0025] FIG. 11 is a flow chart showing a method for determining whether amputation has occurred, in accordance with some embodiments of the present disclosure.

[0026] FIG. 12 is a flow chart showing a method for determining alertness, in accordance with some embodiments of the present disclosure.

[0027] FIG. 13 is a flow chart illustrating an algorithm for determining a triage status of a casualty, in accordance with some embodiments of the present disclosure.

[0028] FIG. 14 is an example illustration of casualty information displayed on a user interface for use by an operator.

[0029] FIG. 15 illustrates a method for performing casualty assessment using a multimodal LLM according to an example embodiment of this disclosure.

[0030] FIG. 16 illustrates a multimodal LLM system for performing casualty assessment according to an example embodiment of this disclosure.

[0031] FIG. 17 illustrates an example multimodal LLM method and system input and output according to an example embodiment of this disclosure, the multimodal method and system shown in FIG. 15 and FIG. 16, respectively.

[0032] FIG. 18 illustrates a decoder only LLM system architecture for performing casualty assessment according to an example embodiment of this disclosure.DETAILED DESCRIPTION

[0033] A more complete understanding of the processes and apparatuses disclosed herein can be obtained by reference to the accompanying drawings. These figures are merely schematic representations based on convenience and the ease of demonstrating the existing art and / or the present development, and are, therefore, not intended to indicate relative size and dimensions of the assemblies or components thereof.

[0034] Although specific terms are used in the following description for the sake of clarity, these terms are intended to refer only to the particular structure of the embodiments selected for illustration in the drawings, and are not intended to define or limit the scope of the disclosure. In the drawings and the following description below, it is to be understood that like numeric designations refer to components of like function.

[0035] Spatially relative terms, such as “beneath,” “below,” “lower,” “above,” “upper” and the like, are intended to encompass different orientations of the device in use or operation in addition to the orientation depicted in the figures. The devices may be otherwise oriented (rotated 90 degrees or at other orientations) and the spatially relative descriptors used herein may likewise be interpreted accordingly.

[0036] Numerical values in the specification and claims of this application should be understood to include numerical values which are the same when reduced to the same number of significant figures and numerical values which differ from the stated value by less than the experimental error of conventional measurement technique of the type described in the present application to determine the value. All ranges disclosed herein are inclusive of the recited endpoint.

[0037] The modifier "about" used in connection with a quantity is inclusive of the stated value and has the meaning dictated by the context (for example, it includes at least the degree of error associated with the measurement of the particular quantity). When used with a specific value, it should also be considered as disclosing that value. For example, the term “about 2” also discloses the value “2” and the range “from about 2 to about 4” also discloses the range “from 2 to 4.” The term “about” may refer to plus or minus 10% of the indicated number.

[0038] The term “drone” is used herein to refer to a self-propelled robot which can travel either through the air or on the ground. Examples of drones include unmanned aerial vehicles (UAVs) and unmanned ground vehicles (UGVs). A drone may be capable of autonomous propulsion, and may accept periodic human commands to perform various tasks.

[0039] The present disclosure relates to Drone Assisted Rapid Triage (DART) systems that support first responder triage decisions following MCls by collecting as much information as possible and utilizing that data to support resource-optimized decisionmaking. Autonomous systems are used to search for, localize, and assess casualties. Standoff sensors on uncrewed ground and aerial autonomous systems will capture the physiological signatures of casualties during an MCI, and through sensor fusion will make an assessment of each casualty’s underlying physiological state. Detection of hard-to- find or obscured casualties can be more easily accomplished through the coordination ofmultiple sensor delivery platforms, enabled by autonomous control. Information regarding casualties can be synthesized, screened for the need for life saving interventions (LSIs), and presented to first responders in a clear and concise format to maximize medical resources and expedite triage decision making.

[0040] Initially, FIG. 1 diagrammatically illustrates a scene 100 of a mass casualty incident (MCI) undergoing initial triage using drones such as unmanned aerial vehicles (UAVs) 102 or unmanned ground vehicles (UGVs) 104. The MCI scene 100 could be the scene of a natural disaster such as an earthquake, hurricane landfall, flood, tsunami landfall, or the like. The MCI scene 100 could alternatively be the scene of a disaster in an urban environment such as a large-scale fire, a building collapse, or so forth. The MCI scene 100 could alternatively be a battlefield. These are merely some nonlimiting illustrative examples.

[0041] As diagrammatically indicated in the upper enlarged UAV image, each UAV 102 includes sensors 106, 108, 110 that provide various sensor signals for locating casualties 112 in the MCI scene 100 and for performing triage of such casualties. Although not shown, each UGV 104 could also include such sensors. By way of some nonlimiting illustrative examples, the sensor 106 may be an acoustic sensor such as a microphone, a phased acoustic array operating in the audible frequency range, a phased ultrasound (US) array operating in ultrasonic frequencies, various combinations thereof, or so forth. The sensor 108 may be a radar, such as an ultrawide-band (UWB) radar. The sensor 110 may be an imaging device such as a video camera, electro-optical (EO) sensor, an infrared (IR) sensor, or combination thereof.

[0042] The sensor data are wirelessly transmitted to a control station 120, which in the illustrative example is a pair of mobile control stations deployed in respective vans or other emergency response vehicles. One or more algorithms are used to perform triage. Operators 114, such as emergency response personnel, can use user interfacing devices 122 to interact with the control station 120. For example, inset 116 illustrates an electronic tablet or cellphone 122.

[0043] FIG. 2 is a flow chart illustrating an illustrative method 200 for performing triage using a DART system. It is noted that not all steps described in the flow chart are required, and other steps may occur that are not described in the flow chart.

[0044] It is contemplated that the triage is needed due to a mass casualty incident (MCI). In step 202 of FIG. 2A, a search strategy is developed. This may include steps such as updating maps (step 204) and identifying areas of high casualty likelihood (step 206). As a result, a computerized map containing known data is created and continuously updated during the triage process.

[0045] Prior information / data concerning the location of the MCI can be obtained. This may include, for example, a general description of the disaster and any known geographic or demographic data (e.g., topographical GIS data, maps, weather conditions, population). The DART system can then digest this information to develop an optimized search strategy. For example, a list of zones within the search area can be developed to be initially prioritized. As one example, casualties of a large fire are more likely to be found distributed throughout the entire search area, whereas casualties of a flood are more likely to be found in locations downstream of the water source. As another example, a location with many tall trees is more likely to need to be searched by UGVs rather than UAVs whose flight paths could be interdicted or whose cameras might not penetrate through foliage. As a third example, a large open park may only need one drone to cover the area, whereas a collapsed building structure may require multiple drones for complete coverage.

[0046] In step 208 of FIG. 2A, search routes are planned for each drone. This may include creating / setting waypoints for each drone to follow along its search route. The search strategy is used to determine the number of UAVs and UGVs to deploy for specific MCls, and how they should be deployed. Waypoints may be weighted based on the likelihood of finding one or multiple casualties. Waypoints could be sorted into three categories: Waypoints Accessible Only by UGVs, Waypoints Accessible Only by UAVs, and Waypoints Accessible by Any Sensor Delivery Platform. This differentiation will allow for the optimization of path planning when routing specific types of drone platforms to specific waypoints. It is noted that steps 202-208 can be performed in an automated manner using computer algorithms along with manual operator input and feedback.

[0047] In some embodiments of the present disclosure, one or more UAVs are programmed to act as “oversight” or “Eye in the Sky” (EITS) platforms. It is contemplated that such oversight platforms will climb to a high operating altitude and orbit the searcharea, and will have a view of a large area. These EITS platform(s) can update any maps using onboard sensors, and populate physical changes to the MCI area indicative of high casualty likelihood. For example, they may be able to identify collapsed buildings, a plane that crash landed, a car wreck, a field that has caught fire, etc. Individual casualties and clusters of casualties may also be identified. This information can be used to populate waypoints for “seeker” platforms to explore further, prioritizing regions with high casualty likelihood. Seeker platforms will include both UAVs and UGVs that are capable of efficiently navigating close to a casualty to assess their physiological state and any injuries using multi-modal standoff sensors. This model allows for efficient coverage of a large area as well as navigating difficult terrain and obstacles to quickly reach casualties.

[0048] Continuing, as indicated in step 210 of FIG. 2A, equipment is physically deployed to the area of the MCI. Referring back to FIG. 1 , such equipment may include, for example, drones such as UAVs 102 and UGVs 104, personnel 114, control stations 120, and user interfaces 122. In optional step 212 of FIG. 2A, one or more standalone Position / Navigation / Timing (PNT) beacons may be deployed to define a search area for the drone(s), and / or to be used as known location references in the event other PNT systems (such as GPS) are not usable.

[0049] In step 214 of FIG. 2A, one or more control stations 120 are set up. In particular embodiments, each UAV 102 and UGV 104 has a dedicated control station, along with other support equipment. It is contemplated the dedicated control station for each drone will collect telemetry and standoff sensor data, and run data processing and detection algorithms specific to the type and role of its linked drone. Each dedicated control station communicates with a central control station, which can also communicate with user interface(s) 122. The central control station will collate the information from each dedicated control station, and run data processing and casualty triage algorithms. This architecture permits the gathering and processing of data both to and from the drones. This node-based architecture also reduces technical risk by segmenting data management and algorithm functions, allowing for parallel development.

[0050] Generally speaking, each control station will include a processor which can execute processing instructions stored in memory connected to the processor. The processor can be variously embodied, such as by a single core processor, a dual coreprocessor (or more generally by a multiple core processor), a digital processor and cooperating math coprocessor, a digital controller, or the like. The memory may be any type of non-transitory computer readable medium such as random access memory (RAM), read only memory (ROM), magnetic disk or tape, optical disk, flash memory, or holographic memory. The processor and memory may be combined in a single chip. Communications devices such as radio controllers, network interfaces, or wireless routers may be used as well.

[0051] In step 216 of FIG. 2A, each drone is initialized. This may include, for example, ensuring communication with its dedicated ground station, performing sensor calibration, and uploading its search route. The drones are then ready for deployment. It is noted that steps 202-216 could be performed in any order at any desired time.

[0052] Continuing, then, in step 218 of FIG. 2A, each drone is deployed to begin searching for casualties. For purposes of this disclosure, the term “casualty” or “patient” is used to refer to any person who may be located within the search area, regardless of whether the person is actually injured or not.

[0053] Each drone desirably has performance characteristics suitable for search and localization. For example, each drone includes a GPS transmitter for tracking its own location, which permits the location of casualties to be marked on a map as well. Each drone should have a minimum operating lifetime, for example of at least one hour. Each drone should have a minimum operating range, both in distance from its control station and possibly in height (for AGVs). For example, an operating range of greater than one kilometer or two kilometers or up to five kilometers or even 10 kilometers is desirable, as is an operating ceiling of up to 300 meters or higher. A high payload capability may be desirable too, for dropping off supplies if desired. Each drone should also include a communications capability, and can be controlled remotely or operate autonomously. UGVs may use treads or wheels for mobility as desired.

[0054] Depending on the operating environment, the drone itself may be built for survivability. For example, in a military theater of operations, the drone might include advanced materials such as reactive armor or other lightweight protection. It may include a booster system to exit the combat area quickly. Advanced countermeasures might be included to defeat radar-guided, heat-guided, or other types of guided munitions. Thedrone may be designed for continued flight after taking some specified amount of damage. Flight time and Time on Station might be improved with the use of solar arrays that would allow for charging in flight. Drones could be deployed from a “mother ship” that releases and recovers smaller drones.

[0055] Generally speaking, as illustrated in FIG. 1 , each drone carries one or more sensors, such as an acoustic sensor 106, a radar 108, and an imaging device 110. Each sensor is appropriate for different measurements. For example, a camera can provide high spatial and temporal resolution over long ranges. An IR sensor is less sensitive to lighting conditions, has long range, and can directly image heat sources like body heat and / or breaths. Acoustic sensors such as speakers and microphones can generate sounds that can penetrate barriers, and enable direct communication with the casualty. Radar can also penetrate barriers, and is relatively insensitive to environmental conditions. One particular example of a radar sensor is the IWR6843ISK Texas instruments mmWave Radar Sensor, which can be used to measure both heart rate and respiration rate (as will be discussed further herein). These sensors can also provide data that can be processed to determine various physiological signatures, as will be discussed further herein.

[0056] In step 220 of FIG. 2A, casualties are identified. This may be done using onboard EO, IR, radar, and / or phased acoustic array sensors that provide data to realtime object detection models. For example, a real-time object detection model (such as YOLO) that is trained to identify people in an image could be used to detect potential casualties in EO and IR data streams. Thermal signatures could also be used to identify humans. Phased acoustic array sensors could identify sounds associated with human activity and distress. Radar can be used to detect breathing and heartbeats. It is contemplated that each casualty will be assigned a de-identifiable code and associated information will be stored in a database and used to prevent duplicative casualty assessment. It is generally contemplated that the standoff sensors on each drone may be most useful when used within a distance of about 15 feet.

[0057] Identification by multiple different sensors increases the confidence that a potential casualty is identified (versus being a “false positive”). In addition, images from multiple perspectives / angles can further increase confidence in casualty identification.For example, a given casualty might be obscured from one side, but visible from the other side. If image processing flags a potential casualty, the path of a drone may be changed to obtain an image from another angle, which would provide further information to more definitively determine whether a casualty is present or not. It is contemplated that false positives would be flagged geospatially to prevent repeated interrogation I effort.

[0058] As each casualty is identified, s / he is assessed and triaged. In step 230 of FIG. 2B, data, such as vital signs, are captured for each casualty. Steps 232-248 describe several different types of data and physiological characteristics that can be captured and determined. It is noted that not all of these steps must be performed. It is contemplated that desirably, the time needed for assessment of each casualty may be reduced to a time of, for example, about 30 seconds to about 60 seconds, or up to about 2 minutes.

[0059] In step 232 of FIG. 2B, the location of the casualty is identified. The location information is logged and used to update maps.

[0060] In optional step 234 of FIG. 2B, the casualty may be engaged to determine their responsiveness. For example, signatures of responsiveness will be measured using arm movement, blink, and speech detection. Signatures of mobility can be used to evaluate whether the casualty is mobile using real-time pose estimation algorithms. This information can help determine if further assessment is required. EO and IR sensors carried by the drone can be used to perform motion capture, and a speaker and microphone can be used to perform audio-based communication. The drone can use an introductory dialogue with the casualty to humanize itself and build trust. After introducing itself as a medical assistant, the drone can ask a series of response-based questions, such as move your arm if you are okay, blink if you understand, are you injured or in pain?, etc. Visual and / or auditory responses (or lack thereof) can be computationally analyzed. For example, a motion capture I pose estimation algorithm can be used to interpret arm movement and blink responses. A speech-to-text based natural language processing (NLP) algorithm can be used to understand verbal responses. Where faces are exposed and detectable, EO and IR can be used to measure blink responses and eyelid movement. In addition, the system can engage the casualty to move into a suitable position, if possible I necessary, for optical face or chest image capture, to allow for accurate image analysis and respiration monitoring (as will be discussed later herein).Language models can be used on recorded audio to extract self-evaluating injury descriptions from the casualty being assessed, for example using open-source pretrained language models like Bidirectional Encoder Representation from Transformers. First responders and system controllers could also be able to perform two-way auditory interactions with the casualty using the microphones and speakers if desired. Method step 252 also refers to determining alertness in the event additional processing is needed from a control station.

[0061] In step 236 of FIG. 2B, the body temperature of the casualty can be measured. This may be done, for example, directly by a thermal or IR sensor.

[0062] In step 238 of FIG. 2B, the casualty is tracked to maintain their identity and their location. This may be done, for example, using an EITS platform as described above. Tracking of all casualties and non-casualty personnel is important. First, ambulatory casualties tend to move and / or congregate during an MCI and maintaining confident identification of previously assessed casualties avoids re-assessment of already-assessed casualties. Secondly, as first responders begin to simultaneously provide treatment, they will need to be identified and tracked to avoid diverting drones to assess them as potential casualties. This data capture is continuously performed, as indicated by arrow 239 of FIG. 2B.

[0063] Some physiological characteristics may be determined by processing of visual images and / or other captured data. It is contemplated that this processing may be performed at a control station. In step 240 of FIG. 2B, the data captured for each casualty is sent to a control station. In step 242 of FIG. 2B, the control station processes the data to determine one or more physiological signatures of a given casualty.

[0064] Some physiological signatures of interest include those in the MARCH acronym. Those include Massive hemorrhage, Airway management, Respiratory, Circulation, and Hypothermia / Head injuries. These signatures indicate a need for immediate LSIs that medics are trained and equipped to deliver in the field.

[0065] In step 244 of FIG. 2B, whether the casualty is hemorrhaging or has hemorrhaged is determined. This may also include an estimate of how much blood loss has occurred. Referring to FIG. 3, signatures of blood loss can be measured via EO and IR imaging and used to evaluate if the casualty is experiencing a massive hemorrhage.Using computer vision techniques, an anatomical region of interest can be defined around the casualty. Instance segmentation can be performed on EO and IR images to identify and mask the casualty. Instance segmentation can be performed using open-source computer vision software. Reflectance spectroscopy is then performed on the masked anatomical region to detect regions of blood. As indicated in FIG. 3, blood has a unique spectral signature in the IR spectrum. With that information, a blood area percentage can be computed to infer percent blood loss and if a massive hemorrhage occurred. This analysis can be performed across multiple frames in time (provided by EO and IR sensors) to determine if blood loss is increasing or remaining stable.

[0066] FIG. 4 shows an example of a non-transitory computer readable medium implemented as a control station or server computer 610 in electronic communication with the drones 102 / 104. For example, each drone 102 / 104 can include a wireless transmitter configured to transmit the images acquired by that drone to the server computer 610. The server computer 610 includes at least one electronic processor 612 and a memory 614. The memory 614 of the server computer 610 stores instructions executable by the at least one electronic processor 612 to perform several different methods, including methods 700, 800, 900. The memory also performs calculations according to different models, such as a Pose Estimation model 616, a Human Parsing model 618, a filter 620 such as a Hue-Saturation-Value (HSV) filter, and / or a Wound Classifier 622. These will be 15ribbedbed in further detail herein.

[0067] FIG. 5 shows an example of one patient blood loss assessment method 700 as a flowchart. The patient blood loss assessment method 700 includes operations 701- 707. Additional operations may also be performed, and not all illustrated operations must be performed. At an operation 701, the images are acquired using the drones 102 / 104. In the patient blood loss assessment method 700, the images are thermal images and / or infrared images, and thus the cameras 110 of the drones 102 / 104 are thermal and / or infrared cameras.

[0068] At an operation 702, the acquired images are segmented. To do so, the acquired images are input to a human parsing model 618 (stored in the memory 614). The acquired images are segmented using the human parsing model 618 to identify limbs of the patient in the acquired images. For example, the human parsing model 618anatomically segments the patient into head, torso, upper limbs, and lower limbs. To do so, the processor 612 uses the human parsing model 618 as a template to isolate the patient from the background in the images. Once the patient has been segmented in the images, the processor 612 uses the human parsing model 618 to match the limbs of the patient with the template provided by the human parsing model 618.

[0069] At an operation 703, an injury to the patient is identified from the segmented images. In some embodiments, a filter 620 is applied to the identified limbs in the segmented images to identify the injury. The filter 620 can be, for example, a Hue- Saturation-Value (HSV) filter. The HSV filter 620 can be tuned or programmed with custom values or ranges indicative of a presence of blood in the images. These custom values or ranges can include, for example, values of (0,1 ), (150,200), (100,150) for H,S,V respectively using a 256 (8-bit) color palette. These values may change depending on different contexts. The HSV filter 620 is specifically tuned for blood detection, and then applied to identify regions of blood surrounding or pooling around the patient. For example, blood has a different IR spectroscopy signature across multiple wavelengths from other fluids. In some embodiments, the injury is identified by applying a machine learning (ML) wound classifier 622 (stored in the memory 614) to the segmented images. The wound classifier 622 is trained with historical injury data, and is configured to analyze the images to identify injuries to the patient.

[0070] At an operation 704, a blood loss of the patient is estimated from the identified injury in the operation 703. The blood loss estimation can be performed for each image. To do so, a body mask 624 of the patient is generated to identify a level of blood in the segmented (thermal or infrared) images using the generated body mask 624. The body mask 624 is configured to flag significant blood loss and wound presence, with thresholds adjustable based on empirical data. In some embodiments, the HSV filter 620 can be applied to the body mask 624 to quantify the blood level by detecting wounds, providing a comprehensive assessment of the injuries. For example, the body mask 624 (when filtered with the HSV filter 620) can quantify the blood loss across frames of the segmented images. The quantified blood loss is identified as an injury (such as a hemorrhage) when the quantified blood loss exceeds a predetermined blood loss threshold. For example, for determining mass hemorrhage from total body area coveredin blood, a relative area of blood vs no blood is calculated within the entire body mask 624 can be calculated for each frame of the segmented images. The relative blood areas across multiple images may then be sorted into percentiles. This can be useful for removing outliers, such as obstructions or artifacts that disrupt individual frames. For example, if a 50th percentile of relative blood area detected in the segmented images is greater than 30%, this means that in 50% of the images, more than 30% of the body is covered in blood, and so the patient is classified as having a massive hemorrhage. In another example, for determining mass hemorrhage from external bleeding or a pool of blood, a relative area of external blood to the body area can be calculated within the entire body mask 624 for each frame of the segmented images. If a 75th percentile of relative blood area detected in the segmented images is greater than 3%, the patient is classified as having a massive hemorrhage. For determining if a body region is of the patient is bloody or wounded, a relative area of blood vs no blood within the entire body mask 624 can be calculated for each frame of the segmented images. If the 80th percentile of relative blood area detected in the segmented images is greater than 3%, the body region is classified as wounded. There are merely examples and should not be construed as limiting.

[0071] At an operation 705, if the patient is determined to have blood loss above the predetermined blood threshold, then an alert is generated for a medical professional to provide treatment (i.e., LSI) to the patient.

[0072] At an optional operation 706, one or more confidence values can be calculated for the generated alerts to determine whether the alert constitutes a false positive or not. The calculated confidence values can be used to update or refine the process of estimating the blood loss 704 (indicated by a feedback arrow 708). At an operation 707, the “positive” or “correct” generated alert is then transmitted to a medical professional to provide treatment (i.e., LSI) to the patient. For example, the alert can be displayed on an electronic processing device (not shown - such as a smartphone, tablet, or computer) operable by the medical professional, or the alert can be transmitted to a medical facility for dispatch of one or more medical professionals to the medical facility to come provide LSI to the patient.

[0073] In step 246 of FIG. 2B, the respiration rate and / or chest movement of the casualty is measured. Video captured by EO and IR sensors can be broken down into individual frames. This corresponds to the AR steps of the MARCH acronym.

[0074] Signatures of respiration can be measured via both chest movement and breath to ensure an accurate measurement independent of environmental conditions. Features are extracted from each frame, and then boosted to increase the signal. Chest movement can be measured using both EO and IR based pixel motion and UWB based time-of-flight (ToF) or Doppler measurements. A region of interest (Rol) model is used to identify the chest. Oscillations in chest movement can then be used to calculate respiration rate (RR). Such methods generally perform well regardless of differences in lighting, clothing, background noise, and / or body position.

[0075] Breath detection can be measured using both IR and acoustic phased arrays. Breath oscillations can also be used to calculate RR. IR images can show the mouth or nostrils, where thermal deviations occur during respiration. Acoustic sensors may permit fixation on audio near the mouth, where exhale / inhale sounds can be processed to extract breath frequency and thus determine the RR. Wheezing could be identified in the frequency spectrum of decomposed audio recordings of breathing. Sucking chest wounds could be identified using similar IR oscillation methods imaged at the thoracic region to identify thermal signatures of cavity airflow in and out of the chest.

[0076] In FIG. 6, a flowchart illustrates one method for non-contact respiration rate detection 400 that captures and analyze thermal intensity fluctuations in the mouth or nose regions of a subject to determine a respiration rate. Additional method steps may also be performed, and not all illustrated method steps must be performed.

[0077] In step 404, thermal images of the subject’s mouth or nose region are captured. As indicated by box 402, the system may employ an infrared sensor to capture those thermal images. As indicated by box 408, facial recognition software may be utilized to accurately identify and isolate the relevant regions of interest (ROI) in each thermal image, ensuring that the system tracks the mouth or nose region and ignores the rest of the image. In step 406, the thermal intensity in the images is tracked to monitor fluctuations over time. FIG. 7A illustrates how the system detects facial landmarks, focusing specifically on the area between the nose and upper lip. Once these landmarksare identified, a tracking point is placed, and the Lucas-Kanade (LK) optical flow algorithm is applied to stabilize the cropped frame, ensuring that minor movements of the subject's face do not interfere with the accurate detection of respiratory fluctuations.

[0078] . In step 410 of FIG. 6, a pixel intensity tracking module tracks the intensity of each pixel in a cluster of pixels across multiple thermal images. As illustrated in FIG. 7B, the same point on the subject is tracked across time through multiple images, allowing the system to monitor thermal intensity fluctuations at the relevant regions continuously. The tracking ensures that the focus remains on the specific points associated with respiratory activity, improving the system’s ability to capture consistent and accurate data over extended periods.

[0079] In step 412 of FIG. 6, the pixel intensities are filtered to remove irrelevant noise and focus on the thermal variations associated with breathing. This filtering may occur within a frequency range of about 0.1 Hz to 0.8 Hz, typical of human respiratory patterns. In step 414, the pixel intensities can be normalized. Then, to refine the data, a plurality of pixels exhibiting large deviations in their frequency spectrum is identified. These deviations are averaged to determine a first respiration rate 416. As illustrated in FIG. 7C, the normalized thermal intensity at the nose is plotted over time. In some embodiments, it is compared to or against normalized ground truth respiration data, which can be collected for example via respiratory inductance plethysmography. This comparison allows the system to assess the accuracy of its respiration rate calculations by cross-referencing them with established respiratory measurement techniques. By analyzing this data, the system can derive an average respiration rate with high precision, further enhancing the reliability of the detection process. As shown in step 418, the system can operate in a continuous feedback loop, ensuring that the system continuously adapts to the subject’s respiratory patterns in real-time.

[0080] A drone may be used to capture thermal images remotely in scenarios where it is impractical to monitor individuals directly. The captured data may be transmitted to a control station for further analysis. The system can also engage with multiple subjects in each image, identifying and tracking each subject separately using facial recognition techniques. This capability allows for simultaneous monitoring of multiple individuals in crowded or large-scale environments.

[0081] Another method 500 for detecting / determining respiration rate is shown in FIG. 8. This video-based method captures video images of the chest area of the subject to detect respiration rate. Additional method steps may also be performed, and not all illustrated method steps must be performed.

[0082] In step 502, video images of the chest area of the subject are captured. The video images are captured at a resolution and for a time period sufficient to track subtle chest movements indicative of respiration. It is noted that the video images can be captured from multiple angles. In step 504, the chest area of the subject is determined for each video image. This may be done, for example, by using a pose detection algorithm, to identify the region of interest (ROI) in each video image. Different body positions and orientations can be accounted for in different images in this manner. FIG. 9A illustrates these steps of ROI detection, where the system identifies and isolates the chest area or other relevant body parts indicative of respiratory movements. Once the ROI is determined, a video stabilization module can be applied, so that minor subject movements do not interfere with the accurate tracking of respiration-related chest movements.

[0083] In step 506 of FIG. 8, motion within the chest area is identified in a respiratory frequency range. In particular embodiments, the respiratory frequency range is from about 0.1 Hz to about 0.8 Hz. This ensures that further processing focuses on chest movements indicative of breathing. In step 508, the movement of the chest area is tracked over successive video frames, or in other words, over time. This may be done, for example, using high contrast pixel tracking and optical flow methods (such as Lucas- Kanade) to measure the displacement and magnitude of the chest’s motion. High- contrast pixels can be identified using corner detection algorithms (such as Shi-Tomasi). By focusing on these high-contrast points, the system enhances precision. In step 510, this data is subsequently analyzed to calculate the respiration rate. FIG. 9B illustrates a graph showing the intensity of chest movement over time, which correlates to the respiration rate. As shown by arrow 512, the system can operate in a continuous feedback loop to maintain accurate respiration monitoring in real-time.

[0084] Again, a drone may be used to capture the video images. The captured data may be transmitted to a control station for further analysis. The system can also engagewith multiple subjects in each image, identifying and tracking each subject separately using facial recognition techniques. As previously discussed, the drone may also engage the subject, for example via speaker, to move the subject into a position that exposes the relevant body regions (mouth, nose, chest) to the cameras for image capture.

[0085] In some embodiments, both methods 400, 500 are used at the same time. For example, the respiration rates determined by the two methods may be averaged to calculate the reported I final respiration rate. This dual-modality approach enhances the accuracy of the system by corroborating thermal and video-based methods of respiration.

[0086] In step 248 of FIG. 2B, the heart rate of the casualty is measured. This corresponds to the C step of the MARCH acronym. Signatures of circulation can be measured via remote photoplethysmography (rPPG) and heart movement. Heart rate (HR) can be extracted using both techniques. As illustrated in FIG. 10, rPPG can be measured using EO and IR imaging of the patient’s face (in scenarios of facial exposure). The face can be identified using a facial detection algorithm, and rPPG can be extracted using pixel modulation analysis of the RGB values.

[0087] In step 250 of FIG. 2B, estimates of wound severity is made. For example, convolutional neural network (CNN) models can be trained on wound images to classify specific superficial injuries (e.g., wounds, burns), and such models can then be used on images obtained from the drones. This corresponds to the H step of the MARCH acronym.

[0088] In particular embodiments, an estimate of whether the casualty / patient has an amputated limb is made. FIG. 11 shows an example of the patient amputation assessment method 800 as a flowchart. The patient amputation assessment method 800 includes operations 801-810. Additional operations may also be performed, and not all illustrated operations must be performed. This method may be performed, for example using the processor 612 of FIG. 4.

[0089] At an operation 801 , the images are acquired using the drones 102 / 104. At an operation 802, one or more limb position estimates for a given limb of the patient are determined from the images by inputting the images to the pose estimation model 616. The pose estimation model 616 comprises a skeletal representation of the patient, and is configured to generate limb landmarks for the patient. The pose estimation model 616can be any suitable model, such as a “ball-and-stick” model which represents a skeletal structure of a typical human being. The estimated limb positions can be tracked across time in real time. The estimated limb angles can be captured across video frames of the images. In a particular embodiment, a limb position estimation is generated for each of the shoulders, elbows, wrists, knees, angles, and hips of the patient using the pose estimation model 616.

[0090] At an operation 803, one or more visibility values of the limb position estimates of the patient in the images are calculated. For example, the limb visibility values are calculated for each frame in the images. If the 50th percentile of the calculated limb visibility values is less than a threshold value of 0.3, then the limb is classified as an amputation. This threshold has a scale ranging from 0 being indicative of the limb not being visible and 1 being indicative of the limb being perfectly visible, with the value of 0.3 being indicative of an amputation (although this value can be any desired value between 0 and 1 ). A threshold for confidence values is set within the pose estimation model 616 to estimate the presence or absence of anatomical structures based on the calculated visibility values.

[0091] At an operation 804, at least one of proximal and distal limb lengths of the limb position estimates of the patient in the images are calculated and compared to corresponding predetermined proximal limb length thresholds and / or predetermined distal limb length thresholds. In some embodiments, one or more ratios between the calculated proximal and distal limb lengths of each limb are calculated, and these ratios are compared to corresponding predetermined proximal and distal limb length thresholds. If the ratios underrun the predetermined threshold value, then this is indicative of an amputation of one or more limbs of the patient. For example, for determining if limb is amputated, a limb length ratios for all frames in the images can be calculated. If the 50th percentile of limb ratios is less than a threshold value 0.4 (again, this threshold value can be any suitable value between 0 and 1 ), then the patient is classified as having an amputation. The processor 612 then assesses the aspect ratios and logical consistency of these models to identify any discrepancies that may indicate missing limbs. High confidence estimations with illogical anatomy are flagged as potential limb loss.

[0092] At an operation 805, a spectral domain of each acquired image can be analyzed. The analyzed spectral domains can be used to further determine the state of amputation of the patient by identifying changes in limb length. To do so, the analysis of spectral domain of each acquired image includes analyzing frequencies of the spectral domains to determine lengths of the limbs of the patient by applying a Fast Fourier Transform (FFT) to the images. For example, a standard deviation of the frequencies can be sampled to determine one or more spikes in the spectral domains. Spikes in the frequencies can be detected to indicate rapid changes in limb lengths. A presence of a spike indicates that the length of a given limb of the patient is not what is expected, and thus the patient may have lost a limb. An absence of a spike is indicative of the patient not having a limb amputated. In some examples, a speed of change of the spikes can be analyzed to determine the amputation (or lack thereof) of the limbs of the patient.

[0093] At an operation 806, a state of amputation of the given limb of the patient can be determined based on the visibility value calculation operation 803, the proximal and distal ratio calculation operation 804, and the spectral domain analysis operation 805. If each of the calculated visibility values, the calculated proximal and distal ratios, and the spikes of the analyzed spectral domain each exceed the corresponding threshold for that criterion, then the patient is determined to have lost a limb due to amputation. This “triple check” of the patient using these three criterion increase accuracy in determining an amputation status of the patient.

[0094] At an operation 807, if the patient is determined to have an amputated limb, then an alert is generated for a medical professional to provide treatment (i.e., LSI) to the patient.

[0095] At an optional operation 808, one or more confidence values can be calculated for the generated alerts to determine whether the alert constitutes a false positive or not. In some examples, a confidence value of 80% can be used, representing that 80% of the frames in the images are needed to not show a given limb to indicate a missing limb (which can eliminate artifacts from camera angles that impact limb detection). The calculated confidence values can be used to update or refine the pose estimation model 616 (indicated by a feedback arrow 809). The pose estimation model 616 is continuously updated based on these findings to ensure accuracy and reliability in real-timeassessments. At an operation 810, the “positive” or “correct” generated alert is then transmitted to a medical professional to provide treatment (i.e., LSI) to the patient. For example, the alert can be a binary output of “missing” or “not missing”, and can be displayed on an electronic processing device (not shown - such as a smartphone, tablet, or other computer) operable by the medical professional, or the alert can be transmitted to a medical facility for dispatch of one or more medical professionals to the medical facility to come provide LSI to the patient.

[0096] In step 252 of FIG. 2B, an estimate of whether the casualty is suffering from a head injury is made. This also corresponds to the H step of the MARCH acronym. In particular embodiments, the alertness of the patient is determined to make this assessment. This method may be performed, for example using the processor 612 of FIG. 4

[0097] FIG. 12 shows an example of the patient alertness assessment method 900 as a flowchart. The patient alertness assessment method 900 includes operations 901-909. Additional operations may also be performed, and not all illustrated operations must be performed.

[0098] At an operation 901 , the drones 102 / 104 acquire the images of the patient with the cameras 110, and the images are transmitted to the server computer 610. The drones 102 / 104 can be moved or rotated in order to acquire a sufficient amount of images. The images can be acquired and received by the server computer 610, in real time. In some specific examples, the images are captured at a rate of about 1 to about 60 frames per second (resulting in 60-3600 frames per minute), which should be sufficient to evaluate movement of the patient from the images.

[0099] At an operation 902, one or more limb angle estimates for a given limb of the patient are determined from the images by inputting the images to a pose estimation model 616 (stored in the memory 614). The pose estimation model 616 is configured to generate pose landmarks for the patient, focusing on critical limb angles such as elbows, knees, and hips to monitor motor alertness in casualties by analyzing critical limb angles and movement patterns, aiding rapid assessment in emergency scenarios. The pose estimation model 616 can be any suitable model, such as a “ball-and-stick” model which represents a skeletal structure of a typical human being, and so forth. Examples ofavailable pose estimation models include MediaPipe, YOLO, and OpenPose, among others. The estimated limb angles can be tracked across time in real time. The estimated limb angles can be captured across multiple images (i.e. video frames). In a particular embodiment, a limb angle estimation is generated for each of the elbows, knees, and hips of the patient using the pose estimation model 616 for each image.

[0100] At an operation 903, the standard deviations of the one or more limb angle estimates are determined across multiples images for each limb angle. This is done using known equations on the processor.

[0101] At an operation 904, a state of movement of the patient can be determined based on the comparing operation 903. The state of movement can correspond to a motor alertness of the patient, and can include both arrhythmic and rhythmic movements of one or more limbs. In some embodiments, three states of movement can be determined: the patient is moving normally (i.e., is healthy and does not need LSI), the patient is moving abnormally (i.e., is injured), or the patient is not moving (i.e., is unconscious or deceased).

[0102] To do so, in one embodiment, the standard deviation for each limb angle is compared to a predetermined threshold. If the standard deviation EXCEEDS the predetermined threshold for the given limb angle, then this suggests that the patient is moving their corresponding limb (and thus does not have a critical injury for that limb). For example, if the standard deviation is greater than 10%, the patient is moving the limb and is likely to be alert. On the other hand, if the standard deviation UNDERRUNS the predetermined threshold, then this suggests that the patient is not moving the corresponding limb, and thus may indicate a more severe injury with loss of consciousness, and so may need an LSI..

[0103] In some embodiments, at an optional operation 905, a spectral domain of each acquired image can be analyzed. The analyzed spectral domains can be used to further determine the state of movement of the patient in the operation 904. To do so, the analysis of the spectral domain of each acquired image includes analyzing frequencies of the spectral domains to determine rhythmic (or arrhythmic) movements of the limbs of the patient by applying a Fast Fourier Transform (FFT) to each image and tracking changes over time. For example, a standard deviation of the frequencies can be sampledto determine one or more spikes in the spectral domains. A presence of a “large” spike is indicative of normal movement of one or more of the limbs of the patient, and thus the patient is healthy. A presence of a “small” spike is indicative of abnormal movement of one or more of the limbs of the patient, and thus the patient is injured. An absence of a spike is indicative of unconsciousness or death of the patient.

[0104] In some embodiments, to generate movement for input, the patient can be prompted to move (e.g., raise their hand responsive to a question). For example, the drones 102 / 104 can include a speaker to broadcast directions to the patient, such as “move your right arm”. Movement of the patient can be monitored to see if the patient responds to the directions.

[0105] At an operation 906, if the patient is determined to not be moving one of their limbs (i.e., has an injury of one of their limbs), then an alert is generated for a medical professional to provide treatment (i.e., LSI) to the patient

[0106] At an optional operation 907, one or more confidence values can be calculated for the generated alerts to determine whether the alert constitutes a false positive or not. The calculated confidence values can be used to update or refine the pose estimation model 616 (indicated by a feedback arrow 909). At an operation 908, the “positive” or “correct” generated alert is then transmitted to a medical professional to provide treatment (i.e., LSI) to the patient. For example, the alert can be displayed on an electronic processing device (not shown - such as a smartphone, tablet, laptop, or other computer) operable by the medical professional, or the alert can be transmitted to a medical facility for dispatch of one or more medical professionals to the medical facility to come provide LSI to the patient.

[0107] In step 253 of FIG. 2B, the physiological signatures determined by the control station(s) from the processed data is sent to the triage algorithm. This seamlessly collates the data from all drone sensors into a centralized algorithm that continually evaluates and prioritizes casualty response needs as the situation evolves. Again, such data processing occurs continuously, as indicated by arrow 257 in FIG. 2B.

[0108] In step 254 of FIG. 2B, available resources are inputted into the triage system. Maximizing the distribution of such resources is one purpose of triage, with the goal of saving as many lives as possible. Such resources may also change over time, forexample as more first responders / equipment arrive. Examples of such resources may include, for example, the number of first responders; the type of first responder (e.g. EMT vs. paramedic vs. MD); numbers of particular types of equipment (e.g. numbers of portable X-ray units, ambulances, units of blood, etc.); and / or the location of equipment.

[0109] In step 260 of FIG. 2B, the triage status of the casualty is determined. Based on the physiological signatures extracted from individual casualties localized during the casualty search, a primary triage algorithm will categorize and prioritize care for the first responders. A core static triage model with an additional dynamic elements layer allows contextual responses with improved accuracy. The DART system’s individual assessment procedures leverage parallel sensor data acquisition, data processing, and understanding of MCI context. By integrating this information into a dynamic resource allocation model, triage can be streamlined, allowing drones and first responders to reach a greater number of casualties per unit of time.

[0110] FIG. 13 is a flowchart indicating one embodiment of a triage algorithm 300. Not all steps need to be used. This particular triage algorithm utilizes the same base triage categories as START and SALT: MINOR, DELAYED, IMMEDIATE, and EXPECTANT / DEAD. Each category may be designated by a specific color designation for easy recognition by a system operator when viewing a user interface. The triage algorithm sort casualties into these categories as quickly as possible. At minimum, each assessed casualty will be designated as falling within one of the triage categories. Time, resourcing, and situation permitting, the drones may continue to monitor casualties and acquire additional data to refine assessments, predict precise injury classifications, and increase triage certainties.

[0111] The system will use established criteria commonly accepted across MCI field triage guidelines to assign categories. Primary physiological signatures, (i.e. , ambulation, mental alertness, respiratory rate, massive hemorrhage, and pulse) will be assessed immediately. Secondary physiological signatures (e.g., body temperature, severe wounds) will also be used to further inform the triage algorithm. Assessments of physiological signatures may be modified by dynamic adjustments. Signatures such as mental alertness (responsiveness) and respiratory rate will be used as references fortriage determinations where established quantitative field triage guidelines exist. Signatures such as major hemorrhage can also be quantified.

[0112] When a primary signature is deemed acceptable or out of bounds, a triage category can be assigned according to the logic used in the triage algorithm. Triage categories are assigned when a definitive category can be assigned (e.g., the casualty is walking and thus has minimal injuries according to established triage protocols). In cases where primary signatures are inconclusive, secondary physiological signatures and dynamic triage elements are used to complete the categorization.

[0113] Many dynamic elements may impede autonomous MCI triage. Sensor data may be affected by environmental factors such as changes in lighting conditions, smoke or fog, and obscuring obstacles. Dynamic obstacles and situational factors such as the presence of first responders may introduce noise and off-target readings from casualties. Lastly, MCI context such as the number and severity of casualties, the type of MCI (e.g., chemical agent, explosion, natural disaster), time since the start of the MCI, and the available response resources all factor into resource management strategies.

[0114] The triage algorithm can take into account such dynamic elements. These parameters can be balanced to optimize triage efficiency. For instance, there is an inverse relationship between the time spent per causality assessment and the total number of casualties that can be assessed per unit of time. While longer assessment periods allow for more data collection and thus increased triage categorization confidence, they may inadvertently postpone urgent assessments for immediate casualties. Additionally, the types of sensors, data processing, and modeling of secondary physiological signatures can be down-selected to emphasize signatures relevant to MCI context or to efficiently expend energy according to onboard power storage and power consumption constraints. Lastly, signature bounds may need adjusted depending on casualty demographics. For instance, some triage guidelines differ depending on casualty age, and some sensors may be sensitive to anatomical differences such as skin color. Tuning these tradeoffs properly will allow for efficient triage that both minimizes the time required per casualty for an accurate assessment and maximizes operational time.

[0115] Two major elements of dynamic triage are contemplated: (1 ) Triage Confidence Filtering and (2) Triage Resource Allocation. The first element, Triage ConfidenceFiltering, is important for tuning the time / accuracy tradeoff. A static time window to assess each causality may not be appropriate especially when acquiring noisy signals or monitoring an evolving process. Thus, each primary signature is gated until a confidence threshold is achieved. This threshold will decay over time such that the most confident decisions can be made quickly whereas more ambiguous cases will be allotted more time for further data acquisition before a decision is made. The initial rate of decay parameter can be determined by fitting models to the data provided to the triage algorithm, and they can be further modified by contextual factors such as the number of identified casualties that have yet to be assessed.

[0116] Extracting confidence values for each primary physiological signature is dependent on the processing method. For instance, neural network classifiers output class probabilities during prediction, whereas a power spectrum analysis model for predicting heart rate may not. A statistical process control (SPC) algorithm can be used to set bounds for different sensor signals to verify stable processes. Beyond monitoring individual sensors for noisy or aberrant values indicative of low confidence predictions, this permits monitoring of correlations among sensors that capture redundant signatures. This process is also amenable to confidence filtering as the control limits can expand or decay across time as the situation demands. Control limits for each signature can be guided by established triage guidelines and the analysis of the data being acquired by the drones.

[0117] The second dynamic element, Triage Resource Allocation, will manage data acquisition for secondary physiological signatures. Simultaneous collecting, processing, and modeling data streams from many signatures compounded with power costs for telemetry can delay subsequent assessments. To minimize extraneous processing, signatures will be split into primary (mandatory) and secondary (optional) processes. However, depending on specific injury profiles, the type of MCI, and the necessary resolution of triage (e.g., predicting the need for specific LSIs) secondary sensors may supply valuable information. Spending additional time and resources beyond that required for categorization into primary triage class will likely be context dependent and be determined by the confidence of any primary signature deviations.

[0118] This triage architecture allows for flexible substitution of primary or secondary physiological signatures as they are developed. For instance, the shock index, the ratio of heart rate to systolic blood pressure which has shown to be a strong predictor of shock following trauma may be used in the triage algorithm as well.

[0119] Referring now to FIG. 13, the triage algorithm 300 illustrates how data for a given casualty may be used to assign a triage status to the casualty. The method starts at step 302. Decision box 304 asks whether the casualty is walking or not. If yes, then at step 306 the casualty can be assigned a triage status of MINOR. If the answer to step 304 is no, then the method passes to step 308, where primary physiological signatures are evaluated. Those primary physiological signatures may include, for example, the respiratory rate, whether hemorrhage is present, and / or the pulse (i.e. heart rate) of the casualty. Those primary signatures inform decision box 310, which asks whether the casualty is responsive. Regardless of the answer, decision box 312 asks whether all of the primary signatures are within set bounds or not. If the answer is yes, the algorithm continues to decision box 314. If the answer to decision box 312 is no, the algorithm continues to decision box 316.

[0120] Decision box 314 asks whether further assessment is needed for the casualty whose primary signatures are all within bounds. If the answer is no, then in step 320 the casualty is assigned a triage status of DELAYED. This can mean the casualty still has serious medical issues, but is expected to survive until further care can be provided. Put another way, the casualty does not need to be immediately treated to survive. If the answer is yes, then the algorithm continues to box 326.

[0121] Decision box 316 asks whether the casualty is likely to live based on the primary signatures. If the answer is no, then in step 322 the casualty is assigned a triage status of EXPECTANT. This means the casualty is not expected to live, even if life-saving interventions (LSIs) are immediately provided. If the answer is yes, the algorithm continues to decision box 318.

[0122] Decision box 318 asks whether further assessment is needed for the casualty whose primary signatures are out of bounds. If the answer is no, then in step 320 the casualty is assigned a triage status of IMMEDIATE. This means the casualty has a goodchance of surviving if (LSIs) are immediately provided by first responders. If the answer is yes, then the algorithm continues to box 326.

[0123] In box 326, secondary physiological signatures are evaluated. Those secondary physiological signatures may include, for example, the presence and / or severity of hypothermia, major wounds, severe burns, and / or sucking chest wounds. Arrow 328 indicates the secondary physiological signatures are then further considered by the triage algorithm to determine whether a triage status of DELAYED, IMMEDIATE or EXPECTANT is applied to the casualty. Again, the triage status for each casualty can be continuously assessed, and their status can be changed as appropriate.

[0124] Continuing, in step 262 of FIG. 2B, a triaged casualty list or triage report is created. This list I report sorts all of the identified casualties based on their triage status. Applying autonomous data collection and artificial intelligence decision making to MCls reduces or eliminates the cognitive, communication, and manpower limitations to situational assessments and casualty triage and allows first responders to prioritize high risk casualties that need an LSI.

[0125] With information such as the number of first responders and the location of each casualty and their triage status, the casualty list or triage report can take several different forms. For example, the triage report can be in the form of one list that identifies all casualties with IMMEDIATE status, then DELAYED status, then MINOR status, then EXPECTANT status. Alternatively, the triage report could be in the form of several lists, one for each first responder / team, arranged by triage status and location. Thus, for example, first responder team A would receive a casualty list for an area A that sorts the casualties in that area by triage status, first responder team B would receive a different casualty list for area B, etc.

[0126] In step 264 of FIG. 2B, the triaged casualty list or triage report is communicated to one or more operators. This could be done, for example, using printed handouts or other user interface, such as a cellphone, tablet, heads-up display, etc. For example, the Android Team Awareness Kit (ATAK) app can act as the core of the user interface (Ul). The ATAK architecture has a plug-in functionality to integrate data streams into a top- down spatial representation that will give first responders a birds-eye representation of the mass casualty environment. Data streams from the onboard processing on thedrones and the off-board processing done at the control stations can be processed and represented by ATAK as different icons for, for example, casualties, non-casualty humans, potential environmental threats, or notable objects in the mass casualty environment. This provides users with categorization (e.g., casualty that needs LSI vs. stable casualty), with the ability for users to select an individual casualty to get an in-depth readout of the underlying physiological signatures. Whenever the algorithm detects a need for an LSI, that casualty location can be reported immediately to the user with a pop-up describing the requisite LSI and urgency. The resulting user interface can combine existing GPS and geospatial map data with an overlay for real-time situational awareness of ongoing events.

[0127] The user interface can display a real-time map where UAVs navigating a simulation course are represented. As UAVs assess casualties, the map can be updated with blinking icons at each casualty location. Upon completion of an assessment, each icon becomes clickable, revealing detailed casualty reports to first responders. These icons can be color-coded based on triage status. The user interface may also allow for further data input, such as recording administered medical interventions.

[0128] It is contemplated that the user interface could also identify points of injury on the casualty, relevant biological data, administered medication / fluids, and other relevant information. The user interface could also permit the ability for the user to direct the movement or action of one or more drones. For example, video could be directly streamed from a drone to the user interface.

[0129] FIG. 14 is an illustration of a user interface which could be used for providing information to an operator. On the left-hand side is a plan-view map of a search area with several icons that indicate casualties. The color and / or size of each icon could be used to provide information. For example, the color could indicate the triage status, and / or the size could indicate the amount of information available for that particular casualty, or age (actual or estimated), or how much aid is needed by that particular casualty, etc. Each icon can be clicked on to retrieve details of the particular casualty. In the center of the interface, information about the particular casualty can be displayed. For example, the heart rate and respiration rate are provided, along with other icons that indicate other information such as blood loss, eye injury, ability to speak, and / or mobility. A bodydiagram indicates areas of concern, such as the arms. A Notes section provides further specific information on the particular casualty. The right-hand side of the interface provides a video stream of the particular casualty.

[0130] The DART systems improve MCI response in three primary areas. First, information is centralized through the integration of autonomous systems for casualty search, localization, assessment, and triage. Given the complexity and demands of an MCI, effective information integration improves response, assessment, and monitoring. Second, casualty identification and interaction is performed using stand-off sensors. This reduces the need to have a medical responder in close physical contact with each casualty, and increases the speed of initial casualty assessments. Third, an algorithmbased decision-making triage approach minimizes response times for those that need and would benefit the most from immediate help from a first responder. Overall, this should greatly increase the speed and accuracy of triage decisions, and also reduce the cognitive and mental load of first responders. Both clinical and non-clinical factors unique to the MCI can be accounted for. Triage decisions can also be dynamically adjusted as injuries progress and / or as medical resources are made available. The system also assists with intelligent first responder routes and intervention mapping to maximize resource usage. This serves as a force multiplier.

[0131] With reference to FIGS. 16-19, now described are LLM methods and systems for assessing casualties to reduce the response time and improve casualty assessment by providing natural language reports and contextual insights, thereby enhancing first responder decision-making in emergency situations. Specifically, the disclosed technology leverages the capabilities of advanced Large Language Models (LLMs), including visual language models (VLMs), to assist in the assessment of casualties. By processing both visual inputs (images of the casualty) and textual prompts (assessment details), the system utilizes a latent representation of these inputs to interface with an LLM. The output of the LLM is then routed through a custom prediction head, specifically designed to match casualty cards with high accuracy. This prediction head is fine-tuned on data collected from one or more other training sources to ensure it is optimized for triage-specific scenarios. The integration of LLMs allows for the generation of casualty reports in natural language, providing clear, detailed, and contextually relevantinformation to first responders. According to one example embodiment, the LLM methods and systems use the LLAVA NEXT LLM1 , a 7 billion parameter model, demonstrating effective scene understanding and deployment capabilities on field-compatible hardware like the NVIDIA 4090. In some embodiments, this system can be used with stand-off sensor data collected from drones deployed in mass casualty scenarios, such as the DART system previously described with reference to FIGS. 1-15. Specifically, according to one nonlimiting example, the LLM casualty assessment methods and systems can run parallel to the casualty tracking processes 230-252 shown in FIG. 2B, where the LLM based casualty assessment method and system provide additional information to a control station to generate a triage casualty list / report 262.

[0132] Current casualty assessment tools often rely on structured data and may lack the ability to provide detailed contextual information that can be crucial during triage. The LLM-based approach described herein represents a significant advancement by incorporating natural language processing to interpret and communicate complex casualty scenarios in a way that is intuitive for first responders. The use of multimodal inputs ensures a comprehensive understanding of the situation, enhancing the accuracy and relevance of the information provided. Additionally, the ability to fine-tune the model on specific triage data allows for tailored responses that are directly applicable to emergency medical scenarios

[0133] The use of LLMs, including the joint modeling of visual and textual representations, has been used for applications that include the use of transformer encoders and / or transformer decoders. See “Attention Is All You Need”, 31 st Conference on Neural Information Processing Systems (NIPS 2017), Long Beach, CA, USA; and “SimVLM: Simple Visual Language Model Pretraining with Weak Supervision”, arxiv.org:2108.10904v3,15 May 2022.

[0134] Multimodal large language models (MLLMs) are large language models (LLMs) that can process and generate multiple types of data, such as text, images, audio, video, and 3D environments. MLLMs use techniques like deep learning, neural networks, and transformers to understand context across these data types. For example, they can understand the sentiment of text, recognize objects in images, and interpret meaning from audio cues. MLLMs can then combine this information to form a holistic understandingand perform tasks like generating image captions or providing descriptions for social media.

[0135] According to the example LLM casualty assessment methods and systems described herein, the use of MLLMs including a transformer encoder and / or decoder arrangement is used. Examples of LLM platforms include, but are not limited to, CHATGPT, CLAUDE, GOOGLE GEMINI, ZAPIER CENTRAL, MICROSOFT COPILOT, etc.

[0136] Tokenization is a process that transforms text and images into latent representations for processing by the LLM. Tokenization reduces computational demands compared to directly processing pixels and enhances the effectiveness and efficiency of the LLM generation process.

[0137] As will be further described below, the disclosed LLM casualty assessment methods and systems are integrated with an image language model, can be adapted / extended to other video language models, allow a custom model to be defined with a deterministic output space (casualty card), can include a multi-head classification for each casualty report category, enable the computation of a natural language output in parallel with the custom model deterministic output, only require the training of a smaller custom model, include lightweight training since the encoder model stays frozen, leverage powerful open source pretrained LLM / VLMs, and fine tune model heads on triagedomain data.

[0138] With reference to FIG. 15, illustrated is a method for performing casualty assessment using a multimodal LLM according to an example embodiment of this disclosure.

[0139] Initially, at step 1410, the LLM is instructed to assess casualties, which includes 1 ) receiving from step 1411 , a LLM textual prompt instruction to identify and characterize injury patterns, and 2) receiving from step 1412, a LLM textual prompt instruction to assign a casualty triage category.

[0140] At step 1420, an Image or Video is inputted to the LLM, including receiving from step 1421, a text prompt provided alongside the visuals to assess specific injuries or acquire other medical information.

[0141] At step 1430, the LLM processes the input data of steps 1410 and 1420, and a hidden state of the LLM is saved for further processing.

[0142] At step 1440, the hidden state is routed through a custom model to classify injury or patient state, for example according to a predetermined list of injury categories used to triage a plurality of casualty events or incidents.

[0143] At step 1441 , the custom model is trained with domain-specific training data, e.g. ground truth casualty medical assessments that includes images with known injury types and / or categories.

[0144] At step 1450, medical classifications of the LLM processed casualty data is displayed to one or more operators / medics.

[0145] At step 1460, the hidden state is routed through the native language prediction head, and at step 1470, a natural language text output describing any injury included within the image or video input to the LLM is displayed to one or more operators / medics.

[0146] With reference to FIG. 16, illustrated is a multimodal LLM system 1500 for performing a casualty assessment method as shown in FIG. 15 according to an example embodiment of this disclosure.

[0147] The LLM Integration shown uses a multimodal LLM or VLM for visual language model to assist in casualty assessment. The LLM system shown has the unique benefits of presenting the casualty report in natural language and the ability to add additional contextual details useful to a first responder. Furthermore, premier versions of these LLM models (CHATGPT, GEMINI, etc...) are becoming more powerful and accessible with time which could greatly enhance predictions. FIG. 16 shows an example architecture of an interface with these models. Operationally, a latent representation (i.e. hidden state) of the inputs (images of the casualty and assessment text prompts) as it passes through the LLM is routed to a custom prediction head 1560. The custom prediction head 1560 matches a casualty card 1570 and the custom prediction head 1560 is fine-tuned on training data from other sources, such as domain specific injury images, etc. Advantageously, the disclosed system take advantage of the powerful open-source models pretrained on large amounts of data while only having to train a lightweight prediction head on triage domain data.

[0148] As shown the LLM system 1500 includes an encoder and decoder LLM architecture 1540.

[0149] A DART or other image capturing device 1510 acquires one or more images, i.e. frames, of a potential casualty including a frame sequence 1511 which are input to the LLM 1540.

[0150] In addition, input to the LLM transformer encoder system 1540 is a first Text Prompt 1520 such as: “You are a combat medic... Assess the casualty...” and input to the LLM decoder system is a second Text Prompt 1530 such as: “The most likely injuries are

[0151] The LLM encoder system 1540 includes a convolution processing stage 1541 which receives the captured casualty images, a positional and patch embedding process (Xp1-Xp9;1-9), a token embedding processing stage 1542 including a positional and text embedding process (Xt1-Xt5;1-5) to tokenize textual prompt 1520, a transformer encoder 1543, a hidden state encoder output 1544, a transformer decoder 1545, a decoder textual prompt input 1530 such as: “The most likely injuries are ....”, and a decoder textual output 1550 such as: “The most likely injuries are head trauma...”. A custom model 1560 uses the hidden state as an input to generate a casualty card 1570.

[0152] It is to be understood that the textual prompt 1520 example provided is only an example, and may be any textual prompt, such as predetermined textual prompt which describes a function of the LLM decoder as it relates to the transformation of the received input images, the predetermined textual prompt representing a command to the LLM to access a casualty associated with the casualty image. In other words, a textual prompt generally synonymous with “You are a combat medic...” and “Assess the casualty...” It is also to be understood that the textual prompt 1530 example provided is only an example, and may be any textual prompt, such as predetermined textual prompt which describes a determination to be made by the transformer decoder 1545 as it relates to the received and tokenized input images and textual prompt 1520. For example, the predetermined textual prompt 1530 is generally synonymous with “The most likely injuries are ....”.

[0153] With reference to FIG. 17, illustrated is an example multimodal LLM method and system input and output according to an example embodiment of this disclosure, the multimodal method and system shown in FIG. 15 and FIG. 16, respectively. The VLM1540 used was a LLA A NEXT LLM. The natural language output 1550 provides good scene understanding and a custom model associated with the VLM 1540 is trained for a casualty card model 1570 on top of the VLM 1540.

[0154] With reference to FIG. 18, illustrated is a decoder only LLM system architecture for performing casualty assessment 1600 according to another example embodiment of this disclosure. As shown, in contrast to the encoder and decoder LLM architecture of FIG 17, this LLM casualty assessment system 1600 includes a decoder only architecture 1645, where the decoder only LLM architecture 1645 includes token inputs 1620, a plurality of decoder blocks 1646A-1646C, a custom model 1660 using a hidden state of a decoder block as an input where the custom model 1660 uses the hidden state to generate a casualty card 1570 as previously described. Each of the decoder blocks 1646A-646C include a feed forward neural network process 1647 and a masked selfattention process 1648.

[0155] In the following, some further embodiments are described.

[0156] In a nonlimiting illustrative embodiment, a method of performing casualty assessment using a multimodal large language model (LLM), the method comprising: using an image capture device, generating a casualty image of a subject determined to be a potential casualty; inputting the casualty image to the LLM, the LLM configured to have a first modal input to receive the inputted casualty image and a second modal input preconfigured to receive a predetermined textual prompt which describes a function of the LLM as it relates to the received input images, the predetermined textual prompt representing a command to the LLM to access a casualty associated with the casualty image; the LLM using a transformer encoder and a transformer decoder to process the casualty image and the predetermined textual prompt to generate a natural language output including a textual description of one or more likely injuries associated with the casualty image, and the LLM transformer encoder outputting a hidden state of the transformer encoder to a custom model trained to generate a casualty report which matches the LLM encoder hidden state output with one of a plurality of casualty categorizations, the hidden state representative of the LLM encoder generated representation of the casualty image and the predetermined textual prompt; and communicating the natural language output textual description of one or more likelyinjuries associated with the casualty image and the custom model generated casualty report to at least one operator.

[0157] In another nonlimiting illustrative embodiment, a drone assisted multimodal large language model (LLM) system for performing casualty assessment, the system comprising: one or more drones, each drone including one or more image capture devices mounted thereto; one or more operator controlled audio and / or display devices; and a control station configured to control and communicate with each of the drones, communicate with each of the operator controlled audio and / or display devices, and operatively communicate with a multimodal large language model (LLM) system for performing casualty assessment based on casualty images captured by the one or more drones, the multimodal large language model (LLM) system executing a computer- implemented method comprising: generating a casualty image of a subject determined to be a potential casualty; inputting the casualty image to the LLM, the LLM configured to have a first modal input to receive the inputted casualty image and a second modal input preconfigured to receive a predetermined textual prompt which describes a function of the LLM as it relates to the received input images, the predetermined textual prompt representing a command to the LLM to access a casualty associated with the casualty image; the LLM using a transformer encoder and a transformer decoder to process the casualty image and the predetermined textual prompt to generate a natural language output including a textual description of one or more likely injuries associated with the casualty image, and the LLM transformer encoder outputting a hidden state of the transformer encoder to a custom model trained to generate a casualty report which matches the LLM encoder hidden state output with one of a plurality of casualty categorizations, the hidden state representative of the LLM encoder generated representation of the casualty image and the predetermined textual prompt; and communicating the natural language output textual description of one or more likely injuries associated with the casualty image and the custom model generated casualty report to at least one of the one or more operator controlled audio and / or display devices.

[0158] In another nonlimiting illustrative embodiment, method for performing casualty assessment using a multimodal large language model (LLM), the method comprising: using an image capture device, generating a casualty image of a subject determined tobe a potential casualty; inputting the casualty image to the LLM, the LLM configured to have a first modal input to receive the inputted casualty image and a second modal input preconfigured to receive a predetermined textual prompt which describes a function of the LLM as it relates to the received input images, the predetermined textual prompt representing a command to the LLM to access a casualty associated with the casualty image; the LLM using a transformer decoder to process the casualty image and the predetermined textual prompt to generate a natural language output including a textual description of one or more likely injuries associated with the casualty image, and the LLM transformer decoder outputting a hidden state of the transformer decoder to a custom model trained to generate a casualty report which matches the LLM decoder hidden state output with one of a plurality of casualty categorizations, the hidden state representative of the LLM decoder generated representation of the casualty image and the predetermined textual prompt; and communicating the natural language output textual description of one or more likely injuries associated with the casualty image and the custom model generated casualty report to at least one operator.

[0159] In another nonlimiting illustrative embodiment, a drone assisted casualty assessment system for performing casualty assessment using a multimodal large language model (LLM), the system comprising: one or more drones, each drone including one or more image capture devices mounted thereto; one or more operator controlled audio and / or display devices; and a control station configured to control and communicate with each of the drones, communicate with each of the operator controlled audio and / or display devices, and operatively communicate with a multimodal large language model (LLM) system for performing casualty assessment based on casualty images captured by the one or more drones, the multimodal large language model (LLM) system executing a computer-implemented method comprising: generating a casualty image of a subject determined to be a potential casualty; inputting the casualty image to the LLM, the LLM configured to have a first modal input to receive the inputted casualty image and a second modal input preconfigured to receive a predetermined textual prompt which describes a function of the LLM as it relates to the received input images, the predetermined textual prompt representing a command to the LLM to access a casualty associated with the casualty image; the LLM using a transformer decoder to process the casualty image andthe predetermined textual prompt to generate a natural language output including a textual description of one or more likely injuries associated with the casualty image, and the LLM transformer decoder outputting a hidden state of the transformer decoder to a custom model trained to generate a casualty report which matches the LLM decoder hidden state output with one of a plurality of casualty categorizations, the hidden state representative of the LLM decoder generated representation of the casualty image and the predetermined textual prompt; and communicating the natural language output textual description of one or more likely injuries associated with the casualty image and the custom model generated casualty report to at least one of the one or more operator controlled audio and / or display devices.

[0160] The present disclosure has been described with reference to several different embodiments. Obviously, modifications and alterations will occur to others upon reading and understanding the preceding detailed description. It is intended that the present disclosure be construed as including all such modifications and alterations insofar as they come within the scope of the appended claims or the equivalents thereof.

Claims

CLAIMS:1 . A method for determining a respiration rate of a subject, comprising: capturing thermal images of a mouth or a nose region of the subject; and processing the thermal images to track thermal intensity fluctuations of the mouth or nose region of the subject over time to determine a first respiration rate.

2. The method of claim 1 , wherein the thermal images are captured by an infrared sensor.

3. The method of claim 1 , further comprising using facial recognition on each captured thermal image to identify the mouth or nose region.

4. The method of claim 1 , wherein the thermal intensity fluctuations are tracked by: tracking a pixel intensity of each pixel in a cluster of pixels over multiple thermal images; identifying a plurality of pixels having large deviations in their frequency spectrum; and averaging a power frequency for the plurality of pixels to determine the first respiration rate.

5. The method of claim 4, further comprising, prior to identifying a plurality of pixels having large deviations in their frequency spectrum: filtering the pixel intensity of each pixel; and normalizing the pixel intensity of each pixel.

6. The method of claim 5, wherein the pixel intensity is filtered to be between a frequency range of about 0.1 Hz to about 0.8 Hz.

7. The method of claim 1 , further comprising engaging the subject to move into a position in which the mouth or nose region of the subject is exposed.

8. The method of claim 1 , wherein the thermal images are captured by a drone.

9. The method of claim 8, further comprising transmitting the thermal images to a control station.

10. The method of claim 1 , wherein the thermal images include a plurality of subjects, and further comprising identifying and tracking each subject separately.11 . The method of claim 1 , further comprising: capturing video images of a chest area of the subject; tracking movement of the chest area over time to determine a second respiration rate; and averaging the first respiration rate and the second respiration rate to determine the respiration rate.

12. A system for non-contact respiration rate detection, comprising: a drone comprising an infrared thermal imaging sensor; and a processor configured to include: a facial recognition module for identifying a mouth or a nose region of a subject in thermal images received from the drone; an optical flow module for tracking the mouth or nose region of the subject over time; and a frequency analysis module for isolating respiratory signals.

13. The system of claim 12, further comprising a control station containing the processor and further configured to receive the thermal images from the drone.

14. The system of claim 12, wherein the facial recognition module is further configured to determine whether the thermal images include a plurality of subjects and to separately track each subject.

15. The system of claim 12, wherein the frequency analysis module is further configured to:track a pixel intensity of each pixel in a cluster of pixels over multiple thermal images; filter the pixel intensity of each pixel; normalize the pixel intensity of each pixel; identify a plurality of pixels having large deviations in their frequency spectrum; and average a power frequency for the plurality of pixels to determine a respiration rate of the subject.

16. A method for determining a respiration rate of a subject, comprising: capturing video images of a chest area of the subject; processing the video images to track movement of the chest area over time to determine a first respiration rate.

17. The method of claim 16, wherein the video images are taken from multiple angles to account for different body positions and orientations.

18. The method of claim 16, wherein the video images are processed by: determining the chest area in each video image; identifying motion within the chest area in a respiratory frequency range across the video images; tracking the motion across the video images to determine the first respiration rate.

19. The method of claim 18, wherein the chest area in each video image is determined using a pose detection algorithm.

20. The method of claim 18, wherein the respiratory frequency range is from about 0.1 Hz to about 0.8 Hz.21 . The method of claim 18, wherein the motion within the chest area is tracked by following high contrast pixels.

22. The method of claim 16, further comprising engaging the subject to move into a position in which the chest area of the subject is exposed.

23. The method of claim 16, wherein the video images are captured by a drone.

24. The method of claim 23, further comprising transmitting the video images to a control station.

25. The method of claim 16, wherein the video images include a plurality of subjects, and further comprising identifying and tracking each subject separately.

26. The method of claim 16, further comprising: capturing thermal images of a mouth or a nose region of the subject; and processing the thermal images to track thermal intensity fluctuations of the mouth or nose region of the subject over time to determine a second respiration rate; and averaging the first respiration rate and the second respiration rate to determine the respiration rate.

27. A system for non-contact respiration rate detection, comprising: a drone comprising a video camera; and a processor configured to include: a pose detection module for identifying a chest area of a subject in video images received from the drone; an optical flow algorithm to track motion of the chest area of the subject over time; and a signal processing module for determining a respiration rate of the subject.

28. The system of claim 27, further comprising a control station containing the processor and further configured to receive the video images from the drone.

29. The system of claim 27, wherein the processor further includes a facial recognition module configured to determine whether the video images include a plurality of subjects and to separately track each subject.

30. A patient alertness assessment method, comprising: determining one or more limb angle estimates for a given limb of an associated patient; comparing the determined one or more limb angle estimates to at least one corresponding predetermined angle threshold for the given limb; determining a state of movement of the associated patient based on the comparing; and generating and transmitting an alert based on the determined state of movement.31 . The method of claim 30, further comprising: acquiring, with one or more cameras attached to drones, images of the associated patient; and determining the one or more limb angle estimates from the acquired images.

32. The method of claim 31 , wherein determining one or more limb angle estimates of an associated patient comprises: inputting the acquired images to a pose estimation model; and determining the one or more limb angle estimates using the pose estimation model.

33. The method of claim 32, wherein the pose estimation model comprises a ball and stick model.

34. The method of claim 32, further comprising: calculating one or more confidence values of the generated alert; and updating the pose estimation model using the calculated one or more confidence values.

35. The method of claim 31 , further comprising: analyzing a spectral domain of each acquired image; andidentifying movement of limbs of the patient based on the analyzed spectral domain.

36. The method of claim 35, wherein analyzing the spectral domain of each acquired image comprises: analyzing frequencies of the spectral domains to determine rhythmic movements of the limbs of the patient.

37. The method of claim 36, wherein analyzing frequencies of the spectral domains to determine rhythmic movements of the limbs of the patient comprises: sampling a standard deviation of the frequencies to determine one or more spikes in the spectral domains, the spikes being indicative of movement of one or more of the limbs of the patient.

38. The method of claim 30, wherein determining a state of movement of the associated patient based on the comparing comprises: analyzing standard deviations of the comparing between the determined one or more limb angle estimates to corresponding the predetermined angle thresholds.

39. A patient blood loss assessment method, comprising: segmenting one or more images of an associated patient; identifying an injury to the associated patient from the segmented images; estimating a blood loss of the associated patient from the identified injury; and generating and transmitting an alert based on the estimated blood loss.

40. The method of claim 39, further comprising: acquiring, with one or more cameras attached to drones, images of the associated patient; and segmenting the acquired images.

41. The method of claim 40, wherein segmenting images of an associated patient comprises: inputting the acquired images to a human parsing model; andsegmenting the acquired images to identify limbs of the associated patient in the acquired images using the human parsing model.

42. The method of claim 41 , wherein identifying an injury to the associated patient from the segmented images includes: applying a filter to the identified limbs in the segmented images to identify the injury.

43. The method of claim 42, wherein the filter comprises a Hue-Saturation- Value (HSV) filter.

44. The method of claim 41 , wherein identifying an injury to the associated patient from the segmented images includes: applying a machine learning wound classifier to the segmented images.

45. The method of claim 41 , further comprising: calculating one or more confidence values of the generated alert; and updating the estimating of blood loss using the calculated one or more confidence values.

46. The method of claim 39, wherein estimating a blood loss of the associated patient from the identified injury comprises: generating a body mask of the associated patient; and identifying a level of blood in the segmented images using the generated body mask.

47. The method of claim 46, wherein estimating a blood loss of the associated patient from the identified injury further comprises: applying a Hue, Saturation, Value (HSV) filter to the generated body mask to quantify the blood level.

48. The method of claim 47, wherein identifying a level of blood in the segmented images using the generated full body mask comprises:quantifying a blood level across frames of the segmented images to the generated body mask filtered with the HSV filter; and identifying the quantified blood loss as an injury when the quantified blood loss exceeds a predetermined blood loss threshold.

49. A patient amputation assessment method, the method comprising: determining one or more limb position estimates for a given limb of an associated patient; determining a state of amputation of the given limb of the associated patient based on the one or more limb position estimates; and generating and transmitting an alert based on the determined state of amputation.

50. The method of claim 49, further comprising: acquiring, with one or more cameras attached to drones, one or more images of the associated patient; and determining the state of amputation from the acquired images.

51. The method of claim 50, wherein determining one or more limb angle estimates of an associated patient comprises: inputting the acquired images to a pose estimation model; and determining the one or more limb position estimates using the pose estimation model.

52. The method of claim 51 , wherein the pose estimation model comprises a ball and stick model.

53. The method of claim 51 , further comprising: calculating one or more confidence values of the generated alert; and updating the pose estimation model using the calculated one or more confidence values.

54. The method of claim 49, wherein determining a state of amputation of the associated patient based on the one or more limb position estimates comprises at least one of: calculating one or more visibility values of the one or more limb position estimates; calculating one or more proximal and distal limb lengths of the associated patient; or analyzing a spectral domain of one or more of the acquired images.

55. The method of claim 54, wherein calculating one or more visibility values of the one or more limb position estimates comprises: calculating visibility values of each limb position; and comparing the calculated visibility values of each limb position to corresponding predetermined length thresholds.

56. The method of claim 54, wherein calculating one or more proximal and distal limb lengths of the associated patient comprises: calculating ratios between the calculated proximal and distal limb lengths of each limb; and comparing these calculated proximal and distal limb lengths to corresponding predetermined proximal and distal limb length thresholds.

57. The method of claim 54, wherein analyzing a spectral domain of one or more of the acquired images comprises: analyzing a spectral domain of each acquired image; and identifying changes in limb length from the analyzed spectral domain.

58. The method of claim 57, wherein analyzing the spectral domain of each acquired image comprises: analyzing a rate of changes in frequencies of the spectral domains to determine one or more spikes in the spectral domains, the spikes being indicative of length changes of one or more of the limbs of the associated patient.

59. The method of claim 49, wherein determining a state of amputation of the associated patient based on the one or more limb position estimates comprises each of: calculating one or more visibility values of the one or more limb position estimates; calculating one or more proximal and distal limb lengths of the associated patient; and analyzing a spectral domain of one or more of the acquired images.

60. The method of claim 49, wherein generating and transmitting an alert based on the determined state of amputation comprises: determining whether each of the calculated one or more visibility values, the calculated one or more proximal and distal limb lengths, and the analyzed spectral domains each exceed a corresponding threshold; and generating and transmitting the alert when each of the calculated one or more visibility values, the calculated one or more proximal and distal limb lengths, and the analyzed spectral domains each exceed a corresponding threshold.

61. A method for performing casualty assessment using a multimodal large language model (LLM), the method comprising: using an image capture device, generating a casualty image of a subject determined to be a potential casualty; inputting the casualty image to the LLM, the LLM configured to have a first modal input to receive the inputted casualty image and a second modal input preconfigured to receive a predetermined textual prompt which describes a function of the LLM as it relates to the received input images, the predetermined textual prompt representing a command to the LLM to access a casualty associated with the casualty image; the LLM using a transformer encoder and a transformer decoder to process the casualty image and the predetermined textual prompt to generate a natural language output including a textual description of one or more likely injuries associated with the casualty image, and the LLM transformer encoder outputting a hidden state of the transformer encoder to a custom model trained to generate a casualty report whichmatches the LLM encoder hidden state output with one of a plurality of casualty categorizations, the hidden state representative of the LLM encoder generated representation of the casualty image and the predetermined textual prompt; and communicating the natural language output textual description of one or more likely injuries associated with the casualty image and the custom model generated casualty report to at least one operator.

62. The method of claim 61 , wherein the casualty report indicates a plurality of predetermined categories of possible injuries and the casualty report indicates a prediction of the likelihood of the presence of each category of possible injuries within the casualty image.

63. The method of claim 61 , wherein the image capture device is mounted on a drone, wearable by the at least one operator or part of a portable unit carried by the at least one operator.

64. The method of claim 61 , wherein the casualty image includes a plurality of video frames which are input to the LLM.

65. The method of claim 61 , wherein the LLM is an open source LLM and the custom model is tuned on triage-domain data.

66. The method of claim 61 , wherein the casualty image is captured using one or more image capture devices mounted on a drone; the casualty image from the drone is communicated to a control station for prioritizing a medical response by creating a triaged casualty list using one or both of the natural language output textual description of one or more likely injuries associated with the casualty image and the custom model generated casualty report to at least one operator; and communicating the triaged casualty list to the at least one operator.

67. The method of claim 66, wherein the drone comprises an unmanned aerial vehicle (UAV) or an unmanned ground vehicle (UGV).

68. The method of claim 66, wherein the triaged casualty list is communicated to the at least one operator using one or more of a heads-up display of a helmet and an operator held or wearable portable computer device.

69. The method of claim 61 , wherein the predetermined textual prompt is representative of a natural language expression equivalent to or synonymous with “you are a combat medic” and “access the casualty”, and a second textual prompt is input to the transformer decoder, the second textual prompt representative of a natural language expression equivalent to or synonymous with “the most likely injuries are ....”.

70. A drone assisted multimodal large language model (LLM) system for performing casualty assessment, the system comprising: one or more drones, each drone including one or more image capture devices mounted thereto; one or more operator controlled audio and / or display devices; and a control station configured to control and communicate with each of the drones, communicate with each of the operator controlled audio and / or display devices, and operatively communicate with a multimodal large language model (LLM) system for performing casualty assessment based on casualty images captured by the one or more drones, the multimodal large language model (LLM) system executing a computer- implemented method comprising: generating a casualty image of a subject determined to be a potential casualty; inputting the casualty image to the LLM, the LLM configured to have a first modal input to receive the inputted casualty image and a second modal input preconfigured to receive a predetermined textual prompt which describes a function of the LLM as it relates to the received input images, the predetermined textual prompt representing a command to the LLM to access a casualty associated with the casualty image; the LLM using a transformer encoder and a transformer decoder to process the casualty image and the predetermined textual prompt to generate a natural language output including a textual description of one or more likely injuriesassociated with the casualty image, and the LLM transformer encoder outputting a hidden state of the transformer encoder to a custom model trained to generate a casualty report which matches the LLM encoder hidden state output with one of a plurality of casualty categorizations, the hidden state representative of the LLM encoder generated representation of the casualty image and the predetermined textual prompt; and communicating the natural language output textual description of one or more likely injuries associated with the casualty image and the custom model generated casualty report to at least one of the one or more operator controlled audio and / or display devices.71 . The system of claim 70, wherein the casualty report indicates a plurality of predetermined categories of possible injuries and the casualty report indicates a prediction of the likelihood of the presence of each category of possible injuries within the casualty image.

72. The system of claim 70, wherein the image capture device is mounted on a drone, wearable by the at least one operator or part of a portable unit carried by the at least one operator.

73. The system of claim 70, wherein the casualty image includes a plurality of video frames which are input to the LLM.

74. The system of claim 70, wherein the LLM is an open source LLM and the custom model is tuned on triage-domain data.

75. The system of claim 70, wherein the casualty image from the drone is communicated to the control station for prioritizing a medical response by creating a triaged casualty list using one or both of the natural language output textual description of one or more likely injuries associated with the casualty image and the custom model generated casualty report to at least one operator; andcommunicating the triaged casualty list to the operator controlled audio and / or display devices.

76. A method for performing casualty assessment using a multimodal large language model (LLM), the method comprising: using an image capture device, generating a casualty image of a subject determined to be a potential casualty; inputting the casualty image to the LLM, the LLM configured to have a first modal input to receive the inputted casualty image and a second modal input preconfigured to receive a predetermined textual prompt which describes a function of the LLM as it relates to the received input images, the predetermined textual prompt representing a command to the LLM to access a casualty associated with the casualty image; the LLM using a transformer decoder to process the casualty image and the predetermined textual prompt to generate a natural language output including a textual description of one or more likely injuries associated with the casualty image, and the LLM transformer decoder outputting a hidden state of the transformer decoder to a custom model trained to generate a casualty report which matches the LLM decoder hidden state output with one of a plurality of casualty categorizations, the hidden state representative of the LLM decoder generated representation of the casualty image and the predetermined textual prompt; and communicating the natural language output textual description of one or more likely injuries associated with the casualty image and the custom model generated casualty report to at least one operator.

77. The method of claim 76, wherein the casualty report indicates a plurality of predetermined categories of possible injuries and the casualty report indicates a prediction of the likelihood of the presence of each category of possible injuries within the casualty image.

78. The method of claim 76, wherein the image capture device is mounted on a drone, wearable by the at least one operator or part of a portable unit carried by the at least one operator.

79. The method of claim 76, wherein the casualty image includes a plurality of video frames which are input to the LLM.

80. The method of claim 76, wherein the LLM is an open source LLM and the custom model is tuned on triage-domain data.81 . The method of claim 76, wherein the casualty image is captured using one or more image capture devices mounted on a drone; the casualty image from the drone is communicated to a control station for prioritizing a medical response by creating a triaged casualty list using one or both of the natural language output textual description of one or more likely injuries associated with the casualty image and the custom model generated casualty report to at least one operator; and communicating the triaged casualty list to the at least one operator.

82. A drone assisted casualty assessment system for performing casualty assessment using a multimodal large language model (LLM), the system comprising: one or more drones, each drone including one or more image capture devices mounted thereto; one or more operator controlled audio and / or display devices; and a control station configured to control and communicate with each of the drones, communicate with each of the operator controlled audio and / or display devices, and operatively communicate with a multimodal large language model (LLM) system for performing casualty assessment based on casualty images captured by the one or more drones, the multimodal large language model (LLM) system executing a computer- implemented method comprising: generating a casualty image of a subject determined to be a potential casualty;inputting the casualty image to the LLM, the LLM configured to have a first modal input to receive the inputted casualty image and a second modal input preconfigured to receive a predetermined textual prompt which describes a function of the LLM as it relates to the received input images, the predetermined textual prompt representing a command to the LLM to access a casualty associated with the casualty image; the LLM using a transformer decoder to process the casualty image and the predetermined textual prompt to generate a natural language output including a textual description of one or more likely injuries associated with the casualty image, and the LLM transformer decoder outputting a hidden state of the transformer decoder to a custom model trained to generate a casualty report which matches the LLM decoder hidden state output with one of a plurality of casualty categorizations, the hidden state representative of the LLM decoder generated representation of the casualty image and the predetermined textual prompt; and communicating the natural language output textual description of one or more likely injuries associated with the casualty image and the custom model generated casualty report to at least one of the one or more operator controlled audio and / or display devices.

Images