Life sign detection system, apparatus, and method

Abstract

A Life Sign Detection system (LDS) monitors one or more environments using radar. The radar signals are processed to generate one or more point clouds, each associated with an organism in the environment(s). Life signs for the organisms(s) are inferred based on the radar point clouds. For example, the coordinates of the point cloud may be tracked to determine if the organism is moving about the environment, frequency domain components of the points of the point cloud may be used to determine involuntary movements, or combinations thereof. The LDS may determine a status of the organism(s) based on the inferred life signs. For example, if the organism is moving or has involuntary movement, the organism may be determined to be alive. In some embodiments, the life signs may be compared to normal ranges to determine a status of the organism(s).

Description

LIFE SIGN DETECTION SYSTEM, APPARATUS, AND METHODCROSS REFERENCE TO RELATED APPLICATION

[0001] This application claims the benefit under 35 U.S.C. § 119 of the earlier filing date of U.S. Provisional Application Serial No. 63 / 636,629, titled “Biometric Life Sign Detection System, Apparatus, and Method” filed April 19, 2024, the entire contents of which are hereby incorporated by reference in their entirety for any purpose.BACKGROUND OF THE INVENTION

[0002] There may be a variety of applications where it is useful to perform non-contact vital signs monitoring of an organism. For example, the organism may be in a setting or environment where access to the organism is difficult or there may be security and / or safety concerns about approaching the organism.

[0003] There may be drawbacks to various remote sensing technologies. For example, infrared imaging may be blocked by various obstructions in the environment. Other remote sensing technologies may have difficulty distinguishing between multiple organisms. There may be a need for remote vital sign sensing which operates with high accuracy across a wide range of environments and types of organisms and which can be used to distinguish between different organisms in an environment.BRIEF DESCRIPTION OF DRAWINGS

[0004] Figure 1 is a block diagram of a life sign detection system (LDS) according to some embodiments of the present disclosure.

[0005] Figures 2A and 2B are images of example detectors according to some embodiments of the present disclosure.

[0006] Figure 3 is a block diagram of a detector according to some embodiments of the present disclosure.

[0007] Figure 4 is a flow chart of a method of tracking organisms and inferring their life signs with a LDS according to some embodiments of the present disclosure.

[0008] Figures 5A and 5B are example graphs showing a simplified view of LDS information gathering according to some embodiments of the present disclosure.

[0009] Figure 6 shows an example display that may be presented to a user in some embodiments of the present disclosure.

[0010] Figure 7 is a graph of FMCW data sets according to some embodiments of the present disclosure.

[0011] Figures 8A and 8B are flow charts of processing radar data to generate point cloud information according to some embodiments of the present disclosure.

[0012] Figure 9 is a flow chart of a method of using a LDS according to some embodiments of the present disclosure.DETAILED DESCRIPTION OF THE INVENTION

[0013] The following description of certain embodiments is merely exemplary in nature and is in no way intended to limit the scope of the disclosure or its applications or uses. In the following detailed description of embodiments of the present systems and methods, reference is made to the accompanying drawings which form a part hereof, and which are shown by way of illustration specific embodiments in which the described systems and methods may be practiced. These embodiments are described in sufficient detail to enable those skilled in the art to practice presently disclosed systems and methods, and it is to be understood that other embodiments may be utilized and that structural and logical changes may be made without departing from the spirit and scope of the disclosure. Moreover, for the purpose of clarity, detailed descriptions of certain features will not be discussed when they would be apparent to those with skill in the art so as not to obscure the description of embodiments of the disclosure. The following detailed description is therefore not to be taken in a limiting sense, and the scope of the disclosure is defined only by the appended claims.

[0014] There may be various settings where non-contact and / or remote monitoring of subjects (human and / or non-human) is useful. For example, corrections officers, along with officers and individuals responsible for managing safety and security along with medical, and non-medical personnel are often required to decide whether a subject is alive or deceased within a confined space. This often requires physical touch and waking the subject in order to obtain some level of response, which decreases quality of life or can disrupt a resting subject and can be deemed inhumane. Additionally, often many subjects are confined to a localized area, all of which need to be monitored. This includes incarcerated subjects such as humans or animals, subjects that are voluntarily and non-voluntarily detained and subjects in public gatherings. It may be useful to track the life signs for multiple subjects in an environment.

[0015] The present disclosure is drawn to life sign detection systems, apparatuses, and methods. An example life sign detection system (LDS) includes at least one detector and a reporting device. The detector is positioned in an environment and includes a radar unit. The radar unit directs a radar beam at the environment and collects radar reflections from objects,such as organisms, in the environment which are processed into radar signals. Using the radar signals, the reporting device tracks organisms within the environment. Radar signals from the tracked organisms are used to determine life signs for the tracked organisms, based on movements of the organism. The movements may include voluntary movements (e.g., walking around the environment, exercising, writing, etc.) and involuntary movements such as autonomic movements of various bodily systems (e.g., the movement of the chest while breathing, rapid eye movement, movement of the skin due to the heart contracting, etc.). The movements may change the range of the organisms’ surface (e.g., skin) to the detector, which may be measured by the radar, change the position of the organism with respect to the detector, or combinations thereof. In some embodiments, the movements may be used to derive various metrics such as respiration rate, heart rate, or both, of the tracked organisms. Multiple organisms may be simultaneously tracked within the environment, and multiple types of organisms maybe tracked.

[0016] In some embodiments a LDS may include electromagnetic and optical cross-comparison device using electromagnetic frequencies to conduct millimeter wave measurements and tracking moving objects to detect life signs such as biometrics. The life sign reported information may be audible, digital, visual, or combinations thereof such that when measurements are taken a device will display the interpreted data, either directly on the unit or transported via wired to a distribution system. The system will conduct multi-object detection and scanning for subject observation in a user defined location.

[0017] In an example application, the LDS may determine a number of the tracked organisms which have detectable life signs (or life signs above a threshold). This may determine a number of live organisms in the environment. The number of organisms with life signs is compared to an expected number of organisms, and if there is a difference, an alert may be generated. The use of radar may allow for life signs monitoring even when the organism’s movements are subtle (e.g., they are sleeping or meditating) or if they are obstructed (e.g., under a blanket). In an example implementation, the environment may be a jail or prison cell. The LDS may track prisoners within the cell and determine their life signs. If the number of prisoners with life signs does not match the expected number of prisoners in the cell, an alert may be generated. For example, jail personnel may perform a wellness check on the cell to confirm the status of the prisoner without life signs and / or to render medical aid. Similarly, in some embodiments, the LDS may compare one or more derived metrics, such as heart rate or breathing, to ‘normal’ ranges and signal an alert based on one or more of the derived metrics being outside of the normal range.

[0018] Figure 1 is a block diagram of a LDS according to some embodiments of the present disclosure. The LDS 100 includes one or more detectors 140, each positioned in a respective environment 104, and a reporting device 1 10. In the example of Figure 1 , there are M environments 104, each with one detector 140. However, other example embodiments may include multiple detectors 140 in one or more of the environments 104. For example, extra detectors 140 may be positioned in an environment 104 to increase coverage or reduce blind spots. The reporting device 110 collects information from detector(s) 140 and uses it to track organisms, such as non-human organism 106 and a group of human organisms 108, in the environments 104.

[0019] Each detector 140 includes a radar unit 142 which directs one or more radar beams towards the respective environment 104. For example, the radar units 142 may be FMCW radar units which direct a chirped beam towards the environment 104. The radar unit 142 may have multiple transceivers and may operate in a dual mode multi-input multi-output (MIMO) mode. For example, the radar unit 142 may have 3 transmit antennas and 4 receive antennas in some example embodiments. In other embodiments, the radar unit 142 may have more or fewer transmit and / or receive antennas. In some embodiments, the detector 140 may include an optional camera 144.

[0020] The reporting device 110 has a communications module 114 that sends and receives signals from the detector 140. In some embodiments, the reporting unit 110 may be a general purpose computing device, such as a desk top, lap top, tablet, phone, smart watch or other device. In some embodiments, the reporting device 1 10 may represent a network of devices. For example, in an application where the LDS 100 is used in a correctional facility, then the reporting device 110 may represent the facility’s network of work stations.

[0021] The reporting device 110 includes one or more non-transitory instructions 122 loaded on computer readable media, such as in a memory 120. The reporting device 110 executes the instructions 122 using one or more processors 112 to perform life sign detection. The instructions 122 include box 124 which may cause the LDS 100 to track one or more organisms 106 and / or 108 in the environment 104. For example, the radar information from the radar unit 142 may be processed to generate volumetric information about the environment. The volumetric information may be used to localize or detect one or more organisms 106 and / or 108 based on their movement in the environment 104, shape of their radar reflection in the environment 104, or combinations thereof.

[0022] For example, the box 124 may include processing the radar data from one of the detectors 140 to generate one or more point clouds of objects in the environment 104 wherethat detector 140 is located. The points of the point cloud represent reflections of the radar beam from a surface of the organism. The point clouds may be filtered, for example to remove clutter or noise, or to remove static objects, such as furniture in the environment. The box 124 may include tracking bulk properties of the point cloud, such as the coordinates of a centroid of the point cloud, the shape of the point cloud, other properties, or combinations thereof. These may be used to track gross movement of the organisms 106 and / or 108 in the environment 104. For example, tracking the center of the point cloud may allow for tracking of the motion of the organisms 106 and / or 108 as they move around a room. Tracking the shape of the point cloud may determine a position of the organism 106 and / or 108, for example to determine if they are standing up, sitting, laying down, etc. Further processing may be used in situations, such as the cluster of organisms 108 where the point clouds are closely spaced or overlapping. For example, the LDS 100 may implement one or more processing tools to help distinguish between the point clouds of organisms when they are closely spaced.

[0023] The instructions 122 include box 126 which cause the LDS 100 to infer life signs of the tracked organisms 108. Living organisms, even at their most still, may still have motion components from various involuntary movements. For example, respiration may cause the chest to rise and fall as the lungs pump air, and the heart beat may cause vibration in the skin as the pulse travels through blood vessels, digestion may cause the abdomen to move, rapid eye movement may be detectable in the eyes or eyelids and so forth. These movements may alter the distance between all or a portion of the surface of the organism and the detector 140. The radar unit 142 may measure these changes as changes in range between one or more points of the point cloud which represents the organism to the radar unit 142. The instructions 122 may cause the LDS 100 to analyze the radar information in a frequency domain to extract information about the movement of the organism. For example, one or more filters may be applied to the radar data in the frequency domain to isolate different life sign signals. In an example implementation, band pass filters may be used to separate out different life sign signals. The gross properties of the point cloud may also be used to determine life signs. For example, if the point cloud is moving about the environment 104, it may generally be assumed that the organism 108 is moving under their own power, which may be treated as a sign of life.

[0024] The instructions 122 include box 128 which describes determining the status of the tracked one or more organisms. The inferred life signs may be based on gross movement of the point cloud, involuntary movements, or combinations thereof. In some embodiments, the inferred life signs may be used to determine whether the organism is alive or dead. For example, if they organism 108 is not moving about the environment 104, but they continue tohave movements indicating a pulse and breathing, then the system may designate that organism 108 as being alive. In some embodiments, if the LDS 100 does not detect a living organism in an environment where a living organism is expected, the reporting device 110 may present an alert to a user (e.g., on a screen, via a sound such as an alarm, via text, or combinations thereof).

[0025] In some embodiments, as well as alive or dead, the LDS 100 may also determine one or more health status indicators based on the inferred life signs. For example, in some embodiments, the instructions may cause the LDS system to derive one or more metrics from the life signs, such as vital signs. For example, periodic movement of the chest within a certain frequency range may be used to derive a respiration rate of the organism. Similarly, small periodic movements of the skin within a frequency range may be used to derive a heart rate of the organism. In some embodiments, these vital signs may be compared to thresholds or normal physiological ranges to ensure that they are within an expected range. If they are outside the expected range, then the reporting device 110 may present an alert to the user. For example, the LDS 100 may signal a user if one or more of the organisms 108 is determined to have greatly elevated heart rate.

[0026] In some embodiments, the LDS 100 may combine one or more of the inferred life signs to determine the status of the organism. For example, if one or more of the organisms 106 and / or 108 is determined to have a below normal breathing rate, but the point cloud indicates they are supine on a bed, then an alert may not be necessary, as these life signs may indicate sleep. However, if the organism 106 and / or 108 is lying in bed but with a massively elevated heart rate, it may signal an emergency situation.

[0027] In some embodiments, the detected life signs may be presented to a user. For example, the reporting device may include a display 118, such as a screen, which shows the number of organisms 108 in an environment 104 and whether or not they have life signs. In some embodiments, derived measurements may also be available, such as length of time since last movement above some threshold, heart rate and whether it is in a normal range, breath rate and whether it is in a normal range, or combinations thereof. In some embodiments, the display may list the number of organisms with detectable life signs. In some embodiments, the display 110 may show information for multiple environments 104, multiple organisms within the environments or combinations thereof.

[0028] The instructions 122 may cause the LDS 100 to compare the number of organisms with detectable life signs to an expected number of organisms with life signs in that environment. For example, a user may use an input / output (I / O) system 116 to input a number of expectedorganisms in each environment. If the LDS 100 determines that the number of organisms is different than expected, that one or more of the organisms has no life signs, that one or more of the organisms has life signs below a threshold, or combinations thereof, the LDS 100 may issue an alert. For example, an audible alert may be sounded, a visual alert may be displayed, or combinations thereof. In some embodiments, the alert may be distributed to one or more users. For example, the alert may be sent to user’s smart devices (e.g., smart watch, phone, etc.) to a workstation, to a tablet, to a laptop, or combinations thereof either directly or through a wired or wireless network.

[0029] In some embodiments, the detector may also include optical tracking, for example by including a camera 144. The LDS 100 can be used to determine subject movements such as muscular (voluntary or involuntary) movements in order to determine signs of life in a nonmedical fashion using the radar. The radar data can then be utilized with timed, sequenced optical data collection that can be fused to make determination of life status and crosscorrelated to subject recognition without the correlation of privacy information. The subjects can be correlated to a tracking identification, which is a unique identifier that is capable of correlating the electromagnetic measured information with the optical collected data via a device such as a camera, infrared monitor, acoustic detector, or light measurement device. This system is designed to collect multiple inputs for increased subject life detectability and provide a status of subject life condition.

[0030] While the example of Figure 1 shows certain components and functions in the reporting device, other arrangements may also be used. For example, the detectors may include one or more processors and may perform all or part of the data processing steps to track and determine life signs and compare to an expected number.

[0031] In some embodiments, one or more functions of the reporting device 1 10 may be implemented on the detectors 140 instead, or in addition to being implemented on the reporting device 110. For example, the detectors 140 may have their own processor(s) and memory and implement their own instructions to perform one or more steps. For example, the detector 140 may have instructions which are executed by their processors to direct radar beams to the environment 104. In some embodiments, the detectors 140 may implement one or more filtering or data analysis steps before sending the radar data to the reporting device 110. In some embodiments, the functions of the detector 140 may be implemented using hardware (e.g., filter circuits), software, or a combination thereof.

[0032] The environment may be a correctional facility, a health care facility, an ambulance, a veterinary setting, a school, a government building, a shopping mall, an area with publicgatherings, a security or defense building, an infrastructure location, a transportation hub, a public space, or combinations thereof. For example, the environments may each be a different prison cell, and the tracked organisms may be prisoners within that cell. The reporting device may be loaded with an expected number of prisoners for the different cells, and the LDS system may determine if the number of prisoners with detectable life signs in each cell matches the expected number. If the number is different, an alert may be provided, for example to allow prison personnel to check on the prisoner without life signs.

[0033] The organisms 106 and / or 108 may be selected based on the chosen application. For example, if the application is prisoners in a jail cell, then the human organisms 108 may be monitored but non-human organisms 106 may be ignored. Similarly, if the application is veterinary, then the non-human organism 106 may be monitored but the humans in the environment may be ignored. Tools such as filters based on the size and / or shape of the point cloud may be used to select different types of organism.

[0034] In some embodiments, the LDS 100 may implement a trained machine learning model (MLM) to implement one or more steps of the LDS process. For example, a machine learning model may be presented with radar point cloud information from known targets and trained to recognize the target or some aspect thereof. In an example implementation, the MLM may be presented with a training data set of point cloud data collected from people in all different positions, labeled with those positions (e.g., standing, sitting, lying down, etc.), and then the trained MLM will indicate what position the organism is in based on future radar data. Other examples may include training the MLM to distinguish human organisms from non-human organisms, to distinguish between different activity states (e.g., resting, exercising, reading, talking, pacing, etc.), to distinguish between tightly spaced point clouds representing different organisms, or combinations thereof.

[0035] In some embodiments, the system 100 may be trained to distinguish activities that involve multiple organisms interacting. For example, the system 100 may determine that multiple organisms in close proximity with elevated heart rate represents a fight, and signal an alarm. In some embodiments, the system 100 may distinguish that one or more of the organisms is holding an inorganic object, for example by determining that a portion of the point cloud has no involuntary movements indicating life signs, and signal that the fight involves a weapon.

[0036] In some embodiments, the system 100 may record additional information about the environment 104. For example, the system 100 may track different types of organism in the environment. In an example application where the environment 104 is a jail cell, the system100 may distinguish between inmates and prison staff. For example, the system 100 may interact with a door system which indicates when a prison staff member buzzes into the cell, and track them separately than the other tracked organisms in the cell. In some embodiments, the system 100 may present information on the display 1 18 indicating how many prison staff are in the environment, which prison staff are in the environment, or combinations thereof.

[0037] Figures 2A and 2B are images of example detectors according to some embodiments of the present disclosure. The detectors 200a of Figure 2A or 200b of Figure 2B may be used as the detector 140 in the LDS 100 of Figure 1 . The detector 200a shows an example detector with a camera 206a (e.g., 144 of Figure 1 ). The detector 200b shows an example detector 200b without a camera. In addition, the detector 200b has its faceplate 203b removed. In some embodiments, one type of detector, either 200a or 200b may be used within an environment or across multiple environments. In some embodiments, a mix of detectors may be used. For example, each environment may have one detector unit with a camera such as 200a, and then any additional detector units may lack a camera, such as 200b.

[0038] The detector 200a includes a housing 202a with a radome 204a to house the transceiver of a radar unit (e.g., 142 of Figure 1). The example detector 200a of Figure 2A is mounted on a base and may be free standing. Other example detectors may be mounted in other ways, for example screwed or bolted to a wall or ceiling. The example detector 200a of Figure 2A also includes an optional camera 206a to provide additional information about the environment. The radome 204a and camera 206a may be oriented so they have overlapping fields of view within the environment.

[0039] The detector 200b is shown with its faceplate 203b removed from the housing 202b. The housing includes a cavity which holds a circuit board 208b. The circuit board 208b includes components of the detector such as the radar unit (e.g., 142 of Figure 1) and various logic such as a processor, communications module, or others, used to operate the detector 200b. The detector 200a may also have a similar circuit board, however it is concealed by the housing 202a. The faceplate 203b has a window 204b that aligns with the antenna of the radar unit on the circuit board 208b.

[0040] Figure 3 is a block diagram of a detector according to some embodiments of the present disclosure. The detector 300 may implement the detector 140 of Figure 1 , 200a of Figure 2A and / or 200b of Figure 2B. The detector 300 includes a radar unit 306, which includes one or more antenna 308, a processor 302, a communications module 304, a memory 312, and an optional camera 310.

[0041] The radar unit 306 directs radio frequency energy into an environment the detector 300 is positioned in. The radar unit 306 includes one or more antenna 308 which transmit radio waves, receive radio waves, or both. For example, the radar unit 306 includes a first number of transmit antenna 308 that send radio energy into the environment and a second number of receive antenna 308 that receive radio energy from the environment. The number of transmit and receive antenna may be the same or different.

[0042] The radar unit 306 may be a frequency modulated continuous wave (FMCW) radar unit. In an FMCW radar, the antenna 308 direct one or more radar chirps into the environment.Each chirp involves varying a frequency of the radar beam directed to the environment. An example chirp may involve generating a radar beam with a first frequency and then linearly changing (e.g., increasing) the frequency over time to a second frequency different than the first frequency. Each transmit antenna may generate a number of these chirps during a frame of radar data.

[0043] The radar unit 306 may direct radio frequency energy at a specific frequency or range of frequencies. For example, the radar chirps may be centered around a specific frequency. The frequency may be chosen to interact with the organism’s skin. In some embodiments, the radar unit 306 may direct radar beams at about 60 GHz to the environment. This may be advantageous as it may reflect off water in the skin of the organisms. Other frequencies of radar may be used in other example embodiments, for example frequencies between about 50 GHz to 70 GHz or any other frequency may be used.

[0044] The detector 300 includes a processor 302, communications module 304, and memory 312. The processor 302 and memory 312 may be used to operate the radar unit 306, communications module 304, and optional camera 310. For example, the memory 312 may be loaded with instructions for operating the radar unit 306, such as information about the chirps. In some embodiments, the memory 312 may be loaded with one or more signal processing instructions, to process or analyze the raw data from the radar unit 306.

[0045] The communications module 304 couples the detector 300 to the reporting device (e.g., 110 of Figure 1 ). The communications module 304 may be wired, wireless, or combinations thereof. The communications module 304 transmits radar data, raw, processed, or both, to the reporting device. In some embodiments, the communications module 304 may also receive information from the reporting device, such as instructions to be loaded into the memory 312.

[0046] In some embodiments, the detector 300 includes an optional camera 310. The camera images the environment by capturing light at or around the visible spectrum which strikes a detector of the camera. Similar to the radar data, in some embodiments, the processor 302may analyze or otherwise process the image data from the camera 310. The communications module 304 transmits the image data, raw, processed, or both, to the reporting device.

[0047] Figure 4 is a flow chart of a method of tracking organisms and inferring their life signs with a LDS according to some embodiments of the present disclosure. The flow chart 400 may represent the operation of a LDS such as the LDS 100 of Figure 1 , using detectors such as 200a of Figure 2A, 200b of Figure 2B, and / or 300 of Figure 3. The flow chart 400 represents a simplified workflow. Certain connections between steps, such as feedback loops and interconnections between different parts of the diagram have been omitted for clarity. The steps of the flow chart include a mix of operations that may be operated by hardware and software components. Similarly, the steps of the method of the flow chart 400 may be distributed across the detector (e.g., 140 of Figure 1) and reporting device (e.g., 110 of Figure 1 )-

[0048] The flow chart 400 shows a box 402 which represents a CHIRPset for a radar device. The CHIRPset includes instructions for how to operate a radar unit (e.g., 142 of Figure 1 and / or 306 of Figure 3). For example, the CHIRPset 402 may include instructions about a center frequency of the radar chirps, an upper and lower frequency of the radar chirps, a rate to vary the radar chirp frequency, a duration of the radar chirps, a number of chirps per transmit cycle, or combinations thereof. In some embodiments, the CHIRPset 402 may be loaded on a memory (e.g., 312 of Figure 3) of the detector. Based on the CHIRPset, transmit and receive antennas 404 (e.g., 308 of Figure 3) direct radio frequency energy into an environment (e.g., 104 of Figure 1 ) and receive radio frequency energy which has reflected off an object 406. The received radio frequency energy is passed by the antennas 404 to an analog to digital converter (ADC) 408 which generates digital radar signals. The boxes 402-408 represent steps of various radio frequency (RF) devices of the LDS.

[0049] The ADC 408 provides radar signals to various digital signal processing (DSP) steps of the method flow chart 400. For example, boxes 410-428 generally represent DSP steps. The DSP steps may, in some embodiments, be generally implemented in software of the LDS. The ADC 408 provides the radar data to a fast Fourier transform (FFT) 410 which converts the radar data into a frequency domain. The frequency domain radar data is provided to boxes 412 and 416 which represent an organism counting / tracking chain and to boxes 418-428 which represent a life sign inference chain.

[0050] The FFT 410 provides the frequency domain radar data to an organism counting block 412. The organism counting block 412 splits the radar data into different point clouds representing different objects in the scene. The block 412 may filter out or otherwise removeblocks which represent a static background of the scene (e.g., furniture, walls, etc.) to focus on the objects which may represent organisms. Block 412 is followed by block 416 which describes target tracking. Block 416 involves determining if the point clouds represent organisms and determining bulk properties of the point cloud(s) representing organisms. In some embodiments, the block 416 may include determining coordinates of a point (e.g., a centroid) of the point cloud representing an organism over time. The change in coordinates over time may determine movement of the organism in the environment. In some embodiments, the block 416 may include determining a shape of the point cloud. The shape may reflect an orientation of the organism, for example whether they are standing, sitting, laying down, etc.

[0051] The FFT block 410 also provides the frequency domain radar data to a de-cluttering block 418 which filters the frequency domain radar data to remove ‘clutter’ from the data. For example, the de-cluttering block may apply a noise floor to the frequency domain radar data to remove non-moving objects from the data. The filtered frequency domain radar data is passed to a phase analysis block 420 which uses phase analysis to determine which direction(s) the radar was received from. This may help relate the data to the different object(s) that the data is related to and allow the data to be sorted into sets related to different point clouds.

[0052] Block 420 is followed by block 424 which describes extracting biological movement signals from the radar data. For example, block 422 may include applying one or more filters, such as band-pass filters, to split the frequency domain radar data into different ranges. The band-pass filters have cutoffs chosen based on the expected ranges of different biological signals likely to be present in the organism. For example, a first band pass filter may have upper and lower cutoffs chosen based on an expected highest and lowest possible heart rate, a second band pass filter may have upper and lower cutoffs chosen based on an expected highest and lowest possible breathing rate, a third band pass filter may have upper and lower cutoffs chosen based on the frequency ranges of eye movement, and so forth. The number of filters, and their respective cutoffs, may be chosen based on the number and type of biological signals that the LDS is looking for.

[0053] The filtered data is passed to a constant false alarm rate (CFAR) filter block 424. For each of the ranges split by the filters of block 422, the CFAR filter 424 selects a ‘main’ frequency peak within that range. The filtered data is provided to block 426 which applies a category to each of those peaks. For example, if one of the band-pass filters of block 422 is intended to identify heart rate, then block 422 will select a range of frequency domain signals which may include the heart rate, the CFAR block 424 will narrow that range to the primary peakrepresenting the heart rate within that range, and the categorization block 426 will associate that peak with the organism’s heart rate.

[0054] Block 426 is followed by block 428, which describes categorical analysis. The categorical analysis is application dependent and based on the categorized data generated by the block 426. For example, if one of the categories is heart rate, then the block 428 may include comparing that heart rate to upper and lower limits for a ‘normal’ heart rate to determine if the organism’s heart rate is within the normal range, above the normal range, or below the normal range. In some embodiments, the categorical analysis block 428 may combine one or more other pieces of information. For example, referring to the heart rate analysis, the limits for normal heart rate may be set based on a different factor, such as the movement of the target determined in block 416.

[0055] Blocks 416 and 428 are followed by block 430 which involves synthesizing the analyzed categorical information and the target tracking. For example, a set of categorized biological information and their analysis may be assigned to each tracked organism identified in blocks 412 and 416. For the example the categorized biological information may be related based on the coordinates of the points passed through blocks 418-428 based on the coordinates found in blocks 412-416. The set represents inferred life signs for that organism. For example, block 430 may include combining information to give a heart rate value, breathing rate value, muscle twitch value, etc. to each detected organism in the environment.

[0056] Block 430 may include generating a display to show to a user. For example, the display may show each organism in the environment and categorize them based on the analysis from box 428. Some categories may be binary. For example, if any of the various movement signals are non-zero, the organism may be categorized as ‘alive’, while if all are zero or undetectable, the user interface may signal an alert that the organism should be checked. Some categories may be related to a normal range. For example, the user interface may display a heart rate associated with the organism and whether that heart rate is within a normal range or not. If it is not, the user interface may display a warning such as tachycardia or bradycardia. In some embodiments, the user interface may use different ranges, for example to distinguish between mild tachycardia and a level that requires a wellness check or other intervention.

[0057] In some embodiments, the display may combine one or more pieces of information. For example, a heart rate which is elevated above a normal range may be judged to be normal if the tracking of the point cloud from box 416 indicates that the organism is rapidly moving (e.g., exercising). In another example, if the tracking of box 416 indicates that two organisms are inclose proximity but the categorical analysis box 428 indicates abnormally elevated heart and respiratory rates, it may indicate a fight or other altercation, and the user interface 430 may present an alert. In some embodiments, further analysis may be performed. For example, if a portion of the point cloud is found to not have life signs, it may suggest that the organism is holding an object, such as a weapon.

[0058] In some example implementations, the user interface 430 may compare one or more pieces of information to stored information. For example, if the detector is positioned in a jail cell, then there may be an expected number of organisms in the cell. The user interface 430 may compare the number of organisms tracked in box 416 which have normal life signs as determined in box 428 to an expected number of organisms, and an alert may be raised if the numbers differ.

[0059] Figures 5A and 5B are example graphs showing a simplified view of LDS information gathering according to some embodiments of the present disclosure. Figure 5A shows an example spatial graph 500a representing data gathered by a LDS system across an environment. Figure 5B shows a frequency graph 500b of how life sign data is gathered. The view of Figures 5A and 5B show a simplified example using heart rate as an example life sign which is being determined. The graphs 500a and 500b may represent simplified view of data collected and analyzed by a LDS such as 100 of Figure 1 . For example, the graphs 500a and 500b may be representations of data generated by the flow chart 400 of Figure 4.

[0060] The spatial graph 500a shows a set of range bins and angle bits from a chirp of the radar unit. For example, the range and angle may represent 2D slice of the 3D volume being imaged. Some of the boxes show a number which represents a different peak frequency which was detected in that 2D slice. The boxes may represent portions of a point cloud for an organism. For example, the boxes of graph 500a may represent points of a point cloud determined in boxes 412 and 416 of Figure 4.

[0061] The graph 500b shows a frequency domain plot of the signal for an example one of the boxes of the graph 500a. The frequency domain plots represent filtered data in a range where heart rates are expected which may be generated by the box 426 of Figure 4. For example, the band pass filter in this case selects frequencies between 25 / min and 200 / min since heart rate may generally be expected to fall in that range. In the example of the graph 500b, the point of the point cloud which has its frequency domain data shown has a peak frequency of 81 / min. That peak may be selected based on filtering, such as CFAR filtering of box 424 of Figure 4.

[0062] The legend of the graph 500b shows a total set of data points across the point cloud imaged in graph 500a. In this example, four of the points (e.g., boxes) with detectable heartrate had a heart rate of 72 / min, while ten of the points (e.g., boxes) have a heart rate of 81 / min. Since there are more boxes that show a heart rate of 81 beats per minute, that is judged to be the organism’s heart rate. Other methods of judging the overall heart rate of the organism (e.g., averaging) may be used in other example embodiments.

[0063] Figure 6 shows an example display that may be presented to a user in some embodiments of the present disclosure. The GUI 600 of Figure 6 may, in some embodiments represent the user display of a LDS such as LDS 100. For example, the GUI may represent the user interface of block 430 of Figure 4.

[0064] The GUI 600 shows an example window where 7 example readouts are presented to show how different conditions of a single identified organism may be presented to a user. Each organism, in this example a prisoner, has box that shows their heart rate, whether they are moving, their respiratory rate, and whether a staff member is in the room. Whether or not the prisoner is moving is determined based on overall changes to the coordinates of the point cloud, for example as determined in boxes 412 and 416 of Figure 4. The heart rate and respiratory rate are identified from inferred life signs, such as frequency components of points within the point cloud identified as being part of the prisoner. For example, the heart rate and respiration rate may be determined using steps such as boxes 422-426 of Figure 4. Figures 5A and 5B show an example of how an overall heart rate may be assigned, and the respiration may be identified in an analogous manner.

[0065] The GUI 600 of Figure 6 shows example information, but may represent monitoring multiple organisms in a single environment, or multiple organisms across multiple environments. For example, each of the 7 example organisms may be in a single cell, or in different cells.

[0066] In addition to displaying the raw heart rate and respiration rate, the GUI 600 also displays information about the prisoner’s condition, for example as determined in the categorical analysis step of box 428 of Figure 4. Different indicators are displayed to the user based on the categorical analysis. For example, if the heart rate or respiratory rate is too high or too low, a tringle is displayed behind the readout representing the raw value for that metric. If the value is extremely far outside the normal rage, the triangle may be a different color. In addition, text labels may be shown to represent if the value is regular or irregular. For example, a change in the values over time is determined, and if the value is changing rapidly, the label ‘irregular’ may appear.

[0067] For example, if the heart rate is within a normal range, then the number appears without a background. If the heart rate is outside the normal range, but within an emergency range, then the value is displayed superimposed on a yellow triangle. If the heart rate is outside anemergency range, then the value is displayed superimposed on a red triangle with the word ‘Caution’ below it. The respiratory rate is displayed in a similar fashion. The prisoner’s movement is represented by a binary value and either the word ‘true’ or ‘false’ is shown if movement is detected or not detected respectively. The staff in room indicator shows an ID number for a staff member (e.g., a jail guard, a nurse, etc. who is in the room).

[0068] The first example shows a prisoner with a slightly elevated respiratory rate. Example 2 shows a prisoner with a greatly lowered heart rate. Example 3 shows a subject with an elevated heart rate and an elevated breathing rate. Example 4 shows a subject with a greatly decreased heart rate. Example 5 shows a subject who is holding their breath. The system may be able to recognize this because although the heart rate is normal, the prisoner’s breathing has dropped to 0 and they are not moving (e.g., sitting still). Example 6 shows an example prisoner with decreased respiratory rate. Example 7 shows a prisoner with decreased heart and respiratory rates.

[0069] Figure 7 is a graph of FMCW data sets according to some embodiments of the present disclosure. The graph 700 of Figure 7 is a schematic representation of how a radar unit, such as 142 of Figure 1 and / or 306 of Figure 3 generates a data cube which may then be analyzed to infer life signs. For example, the assembly of the data cube of the graph 700 may represent one or more of the steps of boxes 402-408 of Figure 4.

[0070] The graph 700 includes representations of chirps 710. Each chirp is represented as a triangle, which represents the increasing frequency of the radar pulse over time. In the example implementation of Figure 7, there are three transmit antennas (e.g., 308 of Figure 3) TX0, TX1 , and TX2. Each of the three TX antennas generates and transmits a chirp. There are also four receive antennas (e.g., 308 of Figure 3). Each of the four antennas receives a signal for each of the three transmitted chirps, for a total of 12 received radar signals per transmitted chirp.

[0071] The graph 720 shows a 2D graph which represents the signal received when a chirp is transmitted. The number of columns represent the number of received chirp signals, which is the number of TX antennas times the number of RX antennas. That value is NA, which in this example implementation is 12. The rows represent the range NR, where each box represents the amount of signal representing the reflection of the radar energy in a range bin. In this example there are 10 range bins. The radar unit generates a number of chirps Ncover a ‘frame’ of data. Each frame generates an NAx NRpoints of data. Over Ncchirps, a cube of data which has dimensions NR X NAX NC. The data within this cube may then be analyzed to extract information like point clouds and biological movement signals.

[0072] Figures 8A and 8B are flow charts of processing radar data to generate point cloud information according to some embodiments of the present disclosure. The flow chart 800 is divided up across Figure 8A and 8B. The method represented by the flow chart 800 may be implemented by one or more of the apparatuses, systems, or combinations thereof, described herein. For example, the flow chart may be implemented by the LDS 100 of Figure 1 , such as via the instructions 122 being executed by the processor 112. In some embodiments, one or more steps of the flow chart 800 may be executed on board the detectors 140, on board the reporting device 110 or across a mixture of both devices.

[0073] The flow chart 800 begins with radar chirps 802 (e.g., the chirps 710 of Figure 7). The chirps generate radar data which is passed through an FFT 804 (e.g., 410 of Figure 4). The FFT 804 is performed on a per-chirp and per-antenna basis. The frequency domain data is passed to an averaging process 808, which determines which signals represent static objects in the environment (e.g., furniture, walls, etc.). This static clutter is passed, along with the frequency domain data from box 804 to box 806, which describes removing the static clutter from the data so that the remaining data reflects moving objects in the scene. This information about moving objects forms a data cube 810 (e.g., 730 of Figure 7).

[0074] The filtered data of the data cube 810 is passed through blocks 812-816 to perform moving object detection. Blocks 812 and 814 are repeated for each range bin of the data cube 810 based on azimuth antennas of the radar device. Boxes 812 described spatial covariance estimation and boxes 814 described Capon beamforming. The output of boxes 812 and 814 is a range-azimuth spectrum matrix 816 which represents the data cube 810 filtered and converted into a range-azimuth domain. The range-azimuth spectrum matrix 816 is passed through a filter 818 to extract filtered range and azimuth info of moving objects. The filter 818 may be a 2-pass 1 D CFAR filter. The filtered range and azimuth information generated by the filter 818 forms part of detected object information 820. Boxes 810-818 may form parts of the box 412 of Figure 4.

[0075] The detected object information 820 may be refined using blocks 830 and 840 to perform object tracking (e.g., 416 of Figure 4). For example, blocks 830 and 840 may represent Kalman filtering on the data cube 810 based on the detected objects. The blocks 830 and 840 may represent an iterative process that repeats to update information in the detected object information 820. For example, the blocks 830 and 840 may be repeated to keep updating information such as the coordinates and / or shape of the detected objects. The blocks 830 and 840 may be performed on a point-by-point basis for all the detected points identified by the blocks 812-818 as being parts of an object. The points in the detected objectinformation 820 are fed through blocks 830 and 840 and then updated based on those blocks, and the process is repeated to keep tracking the detected objects.

[0076] Block 830 includes block 832, which describes performing a spatial covariance estimation on the points, block 834 which describes capon beamforming based on a steering matrix. Block 834 generates an elevation spectrum for each point which is part of a detected object. The elevation spectra are passed to block 838, which generates elevation information which is fed back into the detected object information 820 to update the elevation coordinates of the detected objects.

[0077] Block 840 includes block 842 which describes capon beamforming, similar to block 834. Block 842 is followed by block 844 which describes performing a doppler FFT on the points to generate block 846 which describes doppler spectrum information for each of the points determined to be part of a moving object. The doppler spectrum information 846 is fed through block 848 which describes extracting the doppler information which is fed back into the detected object information 820 to update the doppler information of the detected objects. The doppler information generated by block 848 may give information about how the points are moving, and may be used infer life signs (e.g., through boxes 418-428 of Figure 4).

[0078] Blocks 852-858 describe generating static scene information which is also fed into the detected objects data 820. For example, the static scene information may be used to help determine the points of the moving objects in the detected object data 820. Block 852 is generated by box 808 and includes a matrix of antenna and range information for the static clutter. For each range bin and for each antenna, Bartlett beamforming 854 is performed to generate data cube 856. Unlike the data cube 810, the data cube 856 has dimensions of range, azimuth and elevation (e.g., spatial coordinates). The data cube 856 is fed to blocks 858 and 859 which describe filtering and azimuth elevation interpolation respectively. Block 858 may be a 2-pass 1 D CFAR detection filtering scheme in some embodiments. The filtered information from block 858 is combined with the data cube 856 in the interpolation 859 to generate range, azimuth and elevation coordinates for the static scene, which is fed into the detected object data 820.

[0079] Figure 9 is a flow chart of a method of using a LDS according to some embodiments of the present disclosure. The method 900 may be implemented by one or more of the apparatuses, systems, or combinations thereof described herein. For example, the method 900 may be performed by the LDS 100 of Figure 1 . The steps of the method 900 may be performed by a detector (e.g., 140 of Figure 1 , 200a of Figure 2A, 200b of Figure 2B, and / or 300 of Figure 3), by a reporting device (e.g., 1 10 of Figure 1), or a mixture of both.

[0080] The method 900 begins with block 910, which describes directing a radar beam at an environment and receiving radar signals from the environment. The radar beam may be directed from one or more antennas of a detector, and the radar signals may be received by one or more antennas of the detector. In some embodiments, the method 900 may include performing FMCW radar by directing one or more chirps at the environment from the detector. In some embodiments, the method may include directing radar beams with a frequency of between about 60GHz to 64GHz. Other frequency ranges may be used in other example embodiments. In some embodiments, multiple detectors may direct multiple radar beams at multiple environments and receive radar signals therefrom.

[0081] Block 910 may be followed by block 920, which describes tracking an organism based on a point cloud generated from the received radar signals. For the example, the steps 410- 416 of Figure 4 and / or the flow chart 800 of the method of Figure 8 may be used to determine and track the point cloud. In some embodiments, the method may include generating a point cloud for each of multiple organisms in the environment.

[0082] Block 920 may be followed by block 930, which describes inferring life signs for the organism based on the point cloud having at least one movement component. For example, the method 900 may include tracking one or more coordinates of the point cloud over time to determine movement of the organism. The method 900 may include tracking frequency components of the points of the point cloud to determine involuntary movement. In some embodiments, a combination of movement and involuntary movement may be used to infer the life signs. The tracking frequency components may include applying one or more filters, such as band pass filters to frequency domain signals of the points of the point cloud (e.g., blocks 422 and 424 of Figure 4). The filters may select for frequency ranges of involuntary movement such as heart rate, respiratory rate, eye movement, muscle twitch, or combinations thereof.

[0083] Block 930 may be followed by block 940, which describes determining a status of the organism based on the inferred life signs. In some example embodiments, the status may be a binary status such as alive or dead. For example, the method 900 may include determining that the organism is alive if at least one of their inferred life signs is non-zero. For example, if at least one of their movement, one or more involuntary movements, are non-zero, they may be judged to be alive. In some example embodiments, the inferred life signs may be compared to a normal or expected range. For example, the heart rate may be compared to a normal range of heart rates, the breathing rate may be compared to a normal range of breathing rates, or both. In some embodiments, the range may be based on one or more other inferred life signs. For example, if the organism is rapidly moving, then the normal heart rate range may beelevated compared to a stationary organism. In some embodiments, the method 900 may include displaying the determined status of the organism, for example on a display of a reporting device (e.g., 110 of Figure 1 ).

[0084] In some embodiments, the method 900 may include tracking multiple organisms in the environment, each with a respective point cloud generated from the received radar signals, inferring life signs for each of the multiple organisms based on the respective point clouds having at least one movement component, and determining a status for each of the multiple organisms based on the inferred life signs. In some embodiments, the multiple organisms may include humans, non-human animals, or combinations thereof.

[0085] In some embodiments the environment may be a correctional facility, a health care facility, an ambulance, a veterinary setting, a school, a government building, a shopping mall, an area with public gatherings, a security or defense building, an infrastructure location, a transportation hub, a public space, or combinations thereof. For example, the environment(s) may represent jail cells, and the organisms may be prisoners.

[0086] In some embodiments, the method 900 may include comparing a number of organisms with non-zero life signs to an expected number of organisms. The method 900 may include generating an alert (e.g., at a reporting station) if the number of organisms with non-zero life signs does not match the expected number. In some embodiments, the method 900 may include generating an alert if one or more life signs are outside a normal or expected range.

[0087] Of course, it is to be appreciated that any one of the examples, embodiments or processes described herein may be combined with one or more other examples, embodiments and / or processes or be separated and / or performed amongst separate devices or device portions in accordance with the present systems, devices and methods.

[0088] Finally, the above discussion is intended to be merely illustrative of the present system and should not be construed as limiting the appended claims to any particular embodiment or group of embodiments. Thus, while the present system has been described in particular detail with reference to exemplary embodiments, it should also be appreciated that numerous modifications and alternative embodiments may be devised by those having ordinary skill in the art without departing from the broader and intended spirit and scope of the present system as set forth in the claims that follow. Accordingly, the specification and drawings are to be regarded in an illustrative manner and are not intended to limit the scope of the appended claims.

Claims

ClaimsWhat is claimed is:

1. A method comprising: directing a radar beam at an environment and receiving a radar signal from the environment; tracking an organism based on a point cloud generated from the received radar signals; inferring life signs for the organism based on the point cloud having at least one movement component; and determining a status of the organism based on the inferred life signs.

2. The method of claim 1 , further comprising: tracking multiple organisms in the environment, each with a respective point cloud generated from the received radar signals; inferring life signs for each of the multiple organisms based on the respective point clouds having at least one movement component; and determining a status for each of the multiple organisms based on the inferred life signs.

3. The method of claim 2, wherein the multiple organisms include humans, non-human animals, or combinations thereof.

4. The method of claim 1 , further comprising comparing a number of organisms with nonzero life signs to an expected number of organisms in the environment.

5. The method of claim 4, further comprising generating an alert if the number of organisms with the non-zero life signs does not match the expected number of organisms in the environment.

6. The method of claim 4, wherein the environment is a jail cell, the organism is a prisoner, and the expected number of organisms represents a number of prisoners incarcerated in the jail cell.

7. The method of claim 1 , further comprising determining a heart rate, respiratory rate or both for the organism.

8. The method of claim 1 , wherein the environment is a correctional facility, a health care facility, an ambulance, a veterinary setting, a school, a government building, a shopping mall, an area with public gatherings, a security or defense building, an infrastructure location, a transportation hub, a public space, or combinations thereof.

9. The method of claim 1 , further comprising: determining one or more coordinates of the point cloud; and determining motion of the organism based on a change in the coordinates over time; and inferring life signs of the organism based on the determined motion.

10. The method of claim 1 , further comprising: applying one or more band pass filters to frequency components of points of the point cloud, wherein the one or more band pass filters correspond to involuntary movement of the organism; and determining involuntary movement signals of the organism based on the filtered frequency components; and inferring life signs of the organism based on the determined involuntary movement signals.11 . The method of claim 10, further comprising comparing the determined involuntary movement signals of the organism to normal ranges for the involuntary movement signals.

12. The method of claim 1 1 , further comprising generating an alert if one or more of the involuntary movement signals are outside of the normal range.

13. The method of claim 1 1 , further comprising setting the normal range based, at least in part, on one or more other inferred life signs.

14. The method of claim 1 , further comprising removing static object clutter from the received radar signals.

15. A system comprising: a detector comprising:a radar unit configured to direct a radar beam at an environment and receive radar signals from the environment; and a reporting device comprising: a processor; computer readable media loaded with non-transitory instructions which, when executed by the processor cause the reporting device to: track one or more organisms in the environment based on point clouds generated from the received radar signals; infer life signs for each of the tracked one or more organisms based on the point cloud having at least one movement component; and determine a status of each of the one or more organisms based on the inferred life signs.

16. The system of claim 15, wherein the radar unit is configured to generate radar in a frequency range of about 60GHz to about 64 GHz.

17. The system of claim 15, wherein the radar unit is a frequency modulated continuous wave (FMCW) radar unit.

18. The system of claim 15, wherein the non-transitory instructions, when executed by the processor, are further configured to cause the reporting device to compare a number of the one or more organisms with life signs to an expected number of organisms in the environment.

19. The system of claim 18, wherein the reporting device includes a display, and wherein the non-transitory instructions, when executed by the processor, cause the system to display an alert on the display if the number of the one or more organisms with the life signs do not match the expected number of organisms.

20. The system of claim 15, wherein the detector is located in a jail cell and wherein the reporting device is located in a remote location.21 . The system of claim 15, wherein the detector further comprises a camera configured to collect images of the environment.

22. The system of claim 15, further comprising: a plurality of detectors positioned in a plurality of environments, wherein the non-transitory instructions, when executed by the processor, cause the system to track one or more organisms across the plurality of environments.

23. The system of claim 15, wherein the non-transitory instructions, when executed by the processor, cause the system to: apply one or more band pass filters to frequency components of points of the point clouds, wherein the one or more band pass filters correspond to involuntary movement of the organism; determine involuntary movement signals of each of the tracked one or more organisms based on the filtered frequency components; and infer the life signs for each of the one or more organisms based on the determined involuntary movement signals.

24. The system of claim 23, wherein the non-transitory instructions, when executed by the processor, cause the system to compare the determined involuntary movement signals of the one or more organisms to normal ranges for the involuntary movement signals.

25. The system of claim 24, wherein the reporting device comprises a display, and wherein the non-transitory instructions, when executed by the processor, cause the system to display a result of the comparison between the determined involuntary movement signals to the normal range for each of the one or more organisms.

26. A method comprising: directing a radar beam at an environment and receiving radar signals from the environment with a detector; generating one or more point clouds from the received radar signals, each associated with one of one or more organisms in the environment; inferring life signs for each of the one or more organisms based on movement components of the associated one of the one or more point clouds; and displaying a status of each of the one or more organisms based on the inferred life signs on a reporting device.

27. The method of claim 26, further comprising directing a radar beam in a frequency range of about 60GHz to 64 GHz.

28. The method of claim 26, further comprising: determining coordinates for each of the one or more point clouds and determining motion of the associated one of the one or more organisms based on changes in the coordinates; filtering frequency components of points of each of the one or more point clouds to determine involuntary movements of the associated one of the one or more organisms; inferring the life signs based on the determined motion, the determined involuntary movements, or combinations thereof for each of the one or more organisms.

29. The method of claim 28, further comprising determining the status of the one or more organisms based, in part, on a comparison of the determined involuntary movements to a normal range for the involuntary movement.

30. The method of claim 26, further comprising alerting a user based on a comparison between a number of the one or more organisms with life signs compared to an expected number of organisms in the environment.31 . The method of claim 26, further comprising: directing radar beams at a plurality of environments each with a respective detector; and displaying a status of each of the one or more organisms across the plurality of environments.

Images