Non-contact activity detection network for elderly care

Abstract

Determining the person's physical state includes detecting the positions of different parts of the person, converting the detected positions of the person into a point cloud having a density that varies according to the movement of each part, correlating the movement and position data from the point cloud with the positions of known physical states and transitions between different states, and selecting a specific physical state by matching the data from the point cloud with the specific physical state. The positions of the different parts of the person may be detected using a tracking device. The tracking device may be a non-contact tracking device. The tracking device may include at least one wideband radar. The tracking device may wirelessly communicate with at least one server in a cloud computing system. The states may include walking, standing, sitting, lying down, turning over, falling, and / or exiting. [Selected Figure] Figure 1

Description

[Technical field]

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS This application claims priority to U.S. Provisional Patent Application No. 62 / 569,405, entitled “METHOD AND SYSTEM FOR A NON-CONTACT ACTIVITY SENSING NETWORK FOR ELDERLY CARE,” filed October 6, 2017, which is incorporated herein by reference.

[0002] This application is directed to the field of remote activity monitoring using interconnected hardware, software, and cloud components, and in particular to remote monitoring of activities of daily living for older adults using ultra-wideband radar and geometric AI models trained on suitable samples. [Background technology]

[0003] The rapid ageing of the world's population presents challenges of an ever-growing list and scale for our civilization, ranging from dramatic changes in the size and composition of the modern workforce, economics and health insurance, to dramatic shifts in family lifestyles and the structure and costs of elderly care.

[0004] According to the United Nations 2017 World Population Ageing Report, the world population aged 60 years and older is expected to more than double in 2017 compared to 1980, reaching 962 million, and to double again to about 2.1 billion by 2050. Furthermore, the number of people aged 80 years and older is expected to grow even faster than the above rates, tripling from 137 million to 425 million between 2017 and 2050. By 2030, older people (aged 60 years and older) will outnumber children under 10 years, and by 2050 they will outnumber adolescents and young adults aged 10-24 (2.1 billion vs. 2 billion). Proportionately, by 2050, the elderly population is expected to make up 35% of the population in Europe, 28% in North America, and 25% in Latin America.

[0005] Currently, Japan is the oldest country in the world, with 33.4% of the population over 60 years old (4% higher than the second oldest country, Italy, where 29.4% are over 60 years old). Japan is projected to remain the oldest country in the world until at least 2050, when 42.4% of the population will be over 60 years old.

[0006] Population aging in developed countries is the result of two main factors: declining birth rates and increasing life expectancy. Thus, Japan and several other countries with declining overall populations due to low birth rates, some of the highest life expectancies in the world for both men and women, are experiencing rapidly increasing proportions of their aging populations.

[0007] In addition to the ageing world population, there is a significant and growing trend of more and more older people living alone. On average, the proportion of older people living independently (alone or with only a spouse) increased from 24% in 1990 to 37% in 2010. The proportion of older people living independently is much higher in many developed countries, and in "more developed regions", as classified by the United Nations, the proportion of older people living independently approaches 75%, of which 34.5% of women and 17% of men live alone.

[0008] With the above population aging trends in mind, long-term care for the elderly (LTC) has become an issue of national importance in an increasing number of countries. It is estimated that approximately 70% of individuals aged 65 and older will require some type of long-term care services in their lifetime. Governments, corporations, and non-profit organizations around the world offer many long-term care options. In particular, long-term care insurance is becoming an organic part of national health insurance systems (e.g., available through private insurance companies since the late 1970s in the United States and through local governments in Japan since 2000).

[0009] Long-term care providers and services in the United States include adult day care centers, home health agencies, hospices, nursing homes, and residential care communities. According to a 2014 CDC comprehensive survey of long-term care providers, services, and users in the United States, over 6 million individuals aged 65 and older receive assistance from approximately 63,000 of the above providers and services, including 15,600 nursing homes and 30,200 residential care communities, but excluding 4,000 hospices that provide care to an additional 1.34 million individuals. In Japan, approximately 25,000 specialized long-term care facilities provide long-term elderly care.

[0010] One of the most important tasks of long-term care systems for the elderly is the permanent non-invasive monitoring of their condition to ensure a safe stay in long-term care facilities, as well as the proper performance of activities of daily living (ADLs) and the prevention of traumatic injuries such as falls (known to occur in 16% of elderly patients in skilled nursing facilities). Video cameras and other invasive tracking methods cannot be used permanently, as they do not meet the privacy requirements of the elderly. Several solutions utilizing Wi-Fi networks and small radars have been proposed recently with the aim of tracking elderly people and other audiences. Examples include the Radar Health Monitor developed by Kyoto University in collaboration with Panasonic, and the Aura Wi-Fi motion technology from Cognitive Systems Corporation.

[0011] Despite significant progress in the development of non-invasive tracking technologies and systems for the elderly, existing solutions are still rudimentary and do not address the majority of the problems and tasks at hand. Wi-Fi motion technology uses low-frequency signals and receives negative reviews for inaccuracy and failure to detect an individual's status. Radar health monitors focus on monitoring an individual's vital signs, such as heart rate, which requires stationary positioning of the individual, but existing solutions cannot distinguish between walking, standing, and sitting people, so the accuracy and even applicability of measurements at a given moment may be questionable.

[0012] It is therefore desirable to create techniques and systems for comprehensive, non-invasive and intelligent monitoring of the elderly. Summary of the Invention [Means for solving the problem]

[0013] According to the system described herein, determining the physical state of the person includes detecting the positions of different parts of the person, converting the detected positions of the person into a point cloud with density that varies according to the movement of each of the parts, correlating the movement and position data from the point cloud with the positions of known physical states and transitions between different states, and selecting a specific physical state by matching the data from the point cloud with the specific physical state. The positions of the different parts of the person may be detected using a tracking device. The tracking device may be a non-contact tracking device. The tracking device may include at least one wideband radar. The tracking device may wirelessly communicate with at least one server in a cloud computing system. The states may include walking, standing, sitting, lying, rolling over, falling, and / or exiting. A fall may be detected in response to the person transitioning from a walking state to a lying state. During the transition, the person may be detected as having a lower and lower center of gravity of the point cloud. A voice generating device may interact with the person following the fall to confirm the fall and provide basic instructions to the person. Following the voice generating device communicating with the person, a caregiver may be contacted if the person does not move or interact. The voice generating device may include a microphone to receive the voice communication from the person. Each of the states may be associated with a point density, size, orientation, centroid, and bounding box arrangement of the point cloud. Parametric representations of the bounding boxes, point densities, and centroid locations of samples of the different states may be provided as inputs to a neural network classifier. The neural network may be trained by detecting the locations of different parts of the person and providing the neural network to a server of a cloud computing system that receives data from a tracking device that wirelessly communicates with the cloud computing system. The neural network may be a long-short-term memory regression neural network. The neural network classifier may correlate the movement and position data from the point cloud with known physical state locations and may transition between different states to select a particular physical state.Determining the person's physical state may also include maintaining information corresponding to the person's habitual routine state transitions and locations. The habitual routine state transitions and locations may be determined by detecting clusters of points in a multi-dimensional space of sequences of objects, time intervals, locations, and state transitions that represent complex user behavior. In response to the person deviating from the habitual routine state transitions and locations, an alert may be provided to a caregiver. The deviating from the habitual routine state transitions and locations may include cycling around a room for an extended period of time, or repeatedly moving back and forth between two objects. To further classify the behavior and improve detection of dangerous situations, the clusters of points corresponding to the habitual routine may be provided to a cloud computing system for comparison with clusters of points corresponding to other people's habitual routines. The person may be in a room, and objects in the room may be first detected by monitoring absolute coordinates of bounding boxes of the point cloud at various user states. The objects may include a bed, a table, a chair, a bookshelf, a door, and / or a window. The objects in the room may be detected by subtracting multiple positions of the bounding box from the area of ​​the room. The bed may be detected by observing that the person is in a reclining position at a particular height from the floor. The boundaries of a bed may be determined by tracking bounding boxes corresponding to lying down, sitting before or after lying down, and standing before sitting. A bed or couch may be detected by observing adjacent states of standing, sitting, and lying in positions corresponding to the bed or couch. A window may be detected by observing a person standing at the boundary of a room for a relatively long period of time.

[0014] Further, according to the system described herein, a non-transitory computer readable medium stores software for determining a physical state of a person. The software includes executable code for detecting positions of different parts of the person, executable code for converting the detected positions of the person into a point cloud having a density that varies according to the movement of each of the parts, executable code for correlating the movement and position data from the point cloud with positions of known physical states and transitions between different states, and executable code for selecting a particular physical state by matching data from the point cloud with the particular physical state. The positions of the different parts of the person may be detected using a tracking device. The tracking device may be a non-contact tracking device. The tracking device may include at least one wideband radar. The tracking device may wirelessly communicate with at least one server in a cloud computing system. The states may include walking, standing, sitting, lying, rolling over, falling, and / or exiting. A fall may be detected in response to the person transitioning from a walking state to a lying state. During the transition, the person may be detected as having a lower and lower center of gravity of the point cloud. A voice generating device may communicate with the person following a fall to confirm the fall and provide basic instructions to the person. Following the voice generating device communicating with the person, a caregiver may be contacted if the person does not move or interact. The voice generating device may include a microphone to receive voice communication from the person. Each of the states may be associated with a point density, size, orientation, centroid, and bounding box arrangement of the point cloud. Parametric representations of the bounding boxes, point densities, and centroid locations of samples of the different states may be provided as inputs to a neural network classifier. The neural network may be trained by providing the neural network to a server of a cloud computing system that detects the locations of different parts of the person and receives data from a tracking device that wirelessly communicates with the cloud computing system. The neural network may be a long-short-term memory regression neural network.The neural network classifier may correlate the motion and position data from the point cloud with known physical state locations and may transition between different states to select a particular physical state. The software may also include executable code that maintains information corresponding to the person's habitual routine state transitions and locations. The habitual routine state transitions and locations may be determined by detecting clusters of points in a multi-dimensional space of objects, time intervals, locations, and sequences of state transitions that represent complex user behavior. In response to the person deviating from the habitual routine state transitions and locations, an alert may be provided to a caregiver. Deviating from the habitual routine state transitions and locations may include cycling around a room for an extended period of time, or repeatedly moving back and forth between two objects. To further classify the behavior and improve detection of dangerous situations, the clusters of points corresponding to the habitual routine may be provided to a cloud computing system for comparison with clusters of points corresponding to other people's habitual routines. The person may be in a room, and objects within the room may be initially detected by monitoring absolute coordinates of bounding boxes of the point cloud at various user states. The objects may include a bed, a table, a chair, a bookshelf, a door, and / or a window. Objects within a room may be detected by subtracting multiple bounding box positions from the area of ​​the room. A bed may be detected by observing a person lying down at a particular height from the floor. The boundaries of the bed may be determined by tracking bounding boxes corresponding to lying down, sitting before or after lying down, and standing before sitting. A bed or couch may be detected by observing adjacent states of standing, sitting, and lying down in positions corresponding to the bed or couch. A window may be detected by observing a person standing at the boundary of the room for a relatively long period of time.

[0015] The proposed system provides techniques, hardware, software, and systems for persistently tracking an elderly person (user) within one or more rooms, including detecting the user's current state, such as walking, standing, sitting, lying, falling, exiting the room, capturing and monitoring the user's vital signs, such as breathing and heart rate, detecting objects within the room, such as furniture, identifying the user's habitual path and routine, such as falls and significant deviations from habitual routines, confirming dangerous user conditions using voice recordings and automated voice communication with the user, and issuing alerts and warnings to caregivers upon detecting and confirming dangerous and risky conditions. The system is built on compact, always-on devices embedded in rooms and other facilities, including ultra-wideband radar running local software, communicating with cloud-based software, and detecting user conditions using artificial intelligence and machine learning.

[0016] The hardware portion of the system is represented by an always-on tracking device embedded in the room (e.g., connected to an AC outlet or other power source) and includes the following components: 1. One or more ultra-wideband radars, such as Frequency Modulated Continuous Wave (FMCW) radars, with multiple active zones covering the room. 2. A chipset that runs firmware and software for local data processing and other operations. 3. Wireless connectivity to cellular or other WAN or LAN networks for data exchange with the cloud-based portion of the system and over-the-air software and firmware upgrades. 4. Audio recording device. 5. Microphone for communication with user and caregiver. 6. Optionally, a light sensor to measure light intensity and color temperature during the user's sleep period.

[0017] The tracking device placed in the room is designated to monitor a single occupant (user) of the room at all times, tracking the status and performance of the user's ADLs, learning the user's habits, adjusting the tracking model (e.g., modeling the user's gait) and habitual routine, measuring and collecting statistics on the user's vital signs, determining significant deviations from normal conditions, detecting dangerous situations (such as falls or unmotivated wandering around the room), communicating with the user by voice to confirm dangerous conditions and provide immediate advice, and generating warnings and alerts that are communicated to caregivers. Other tasks include generating user status reports and sharing the user status reports with the user and caregivers. Additional designations of the tracking device may include detecting stationary or relatively stationary objects in the room, such as furniture, determining periods during which the user's vital signs can be reliably measured, learning the user model, optimizing the distribution of data exchange and processing tasks between local and cloud software, and upgrading the device's firmware and software. Furthermore, learning results and model parameters may be accumulated in the cloud for the purpose of improving the model and sharing care experiences.

[0018] Tracking devices should not be limited to a single room: for example, multiple devices may be placed in adjacent rooms, bathrooms, etc. to jointly monitor a broader range of user ADLs, routes, and routines.

[0019] The basic object used in dynamic modeling of user states is a point cloud that represents the user's body movements and is generated by the tracking device's radar. The density of the cloud and the accuracy of the representation may depend on various characteristics of the radar. For various commercially available inexpensive radars, the device generates on average 50-120 points for a person walking through a room, and the geometric approximation of the body is restricted to the bounding box of the dynamically changing point cloud. The user point cloud can reflect essentially two categories of user states: moving (e.g., walking or moving in a cart across a room, turning in a bed or chair, performing exercises such as squats) and stationary (e.g., standing, sitting, lying down after lying down on a bed or floor). The density of the user's point cloud in the stationary state can be significantly lower than in the moving state, since the body movements in the stationary state are essentially limited to chest movements due to breathing and heartbeat. The density of the point cloud, along with the bounding box dimensions and location and the centroid of the point cloud, can be used to determine the user state through direct geometric processing, through AI methods such as LSTM (long short-term memory) based regression neural network classifiers, through combined direct / AI methods, etc.

[0020] Specifically, the system may include the following states about the user and the associated properties of the point cloud, its centroids, and bounding boxes:

[0021] W - walking across a room. The point cloud has the highest density compared to the other states and may depend on the direction of movement relative to the radar wavefront. The bounding box has the highest height compared to the other states and the centroid of the bounding box may differ depending on the user's gait.

[0022] St - standing in the center of the room or near an object (chair, table, bookshelf, window, bed, etc.). The point cloud density is low due to the user's stationary position, and the bounding box of the captured point cloud roughly corresponds to the user's chest after inhalation. The point cloud centroid has the highest vertical position (distance from the floor) of all conditions.

[0023] S - Sitting (on a chair, bed, sofa). The bounding box dimensions of the point cloud are similar to those when standing, but the bounding box center of gravity is typically lower, i.e. the point cloud is shifted down by a user-specific height interval depending on seat height and certain body parameters (e.g. leg length).

[0024] L - Lying down (e.g. on a bed sofa). The bounding box dimensions are similar to the corresponding dimensions while standing and sitting, but the bounding box is (i) rotated 90 degrees around the horizontal axis and (ii) raised above the floor by the height of the bed surface.

[0025] T-Turning. The dimensions and positions of the bounding box and centroid are similar to the corresponding dimensions when lying down. Due to the additional translation, the density of the point cloud is higher than in the lying down state.

[0026] F - Fall (lying on the floor after falling). Same as lying down, except the bounding box is at floor level.

[0027] D - Exited, the user is not in the room. If multiple devices are installed in a long-term care facility to track users, they may leave one room and walk to another room, bathroom, patio, etc.

[0028] The above list is not exhaustive of all possible user states - for example, there may be user states specific to bathroom-related ADLs, such as showering, taking a bath, shaving, etc.

[0029] One of the key features of the system is the regular measurement and monitoring of the user's vital signs. Vital signs can only be reliably captured in stationary states (standing, sitting, lying down). Thus, in contrast to conventional solutions, the proposed system is able to detect the optimal period for automatic capture of vital signs by recognizing the user state. The system is also able to distinguish fine-grained vital signs, for example when sitting versus asleep in bed. The methods and software for measuring pulsations representing breathing and heart rate based on radar signal reflections are well known, embedded at firmware level and can be updated over the air if necessary.

[0030] Tracking the user's movements while walking allows the user's gait to be modeled based on the periodic changes in the bounding box positions and the centroid of the point cloud. The unique profile of the user's gait, potentially based on the Fourier transform, can be used for user identification (when multiple people are present in a room) in combination with the bounding box characteristics of other states. The unique profile of the user's gait can also be used to detect warning conditions when the user's gait parameters deviate significantly from normal (e.g. shuffling, unsteady gait, etc.).

[0031] In case the point cloud detected by the tracking device identifies multiple people in the room, for example if the cloud is divided into two or more distinct clusters of points indicating the presence of nurses, visitors or other people, the system can process the acquired information in various ways. In guest mode, the system pauses monitoring of the user status and accordingly stops measuring vital signs (which require a determination of stationary state as described herein) until the master user appears alone in the room. A more advanced option can identify the master user based on the intrinsic parameters of the point cloud and the dynamics of the point cloud (size, gait parameters, etc.) and continue monitoring the master user, ignoring other individuals in the room.

[0032] The system's oriented state graph has user states as vertices, while the edges of the graph are executable transitions from one state to another. Not all transitions are executable; for example, a user cannot start walking immediately after lying down on his bed, or stand near a window immediately after entering a room after leaving.

[0033] All possible transitions between user states define a basic user route, such as entering the room and walking to a chair, standing near the chair, sitting on the chair, walking to the bed, standing near the bed, sitting on the bed, lying on the bed, etc. Most transitions between states have a specific transition procedure assigned to them. Such transition procedures are verified by the system to improve the accuracy of user state detection. For example, transitioning from a walking state to a standing state requires the user to slow down. In another example, if a user is walking across a room and suddenly falls, the user's body will bend forward or backward abruptly. Thus, the bounding box of the intermediate point cloud captured during the transition from the walking state to the falling state will gradually decrease the bounding box height, obtain a lower position of the center of gravity, and increase the length. Such transition procedures may be subject to training of an LSTM RNN model, and the resulting classifier can use these procedures to verify subsequent user states. Alternatively, or in addition to machine learning, transition procedures can be identified by directly processing the geometry of the point cloud and its bounding box.

[0034] Basic user routes can be further combined into habitual routines, which can be formally represented as parameterized paths on a state graph, where parameters include the objects in the room, the time intervals the user spent in each state, etc. Examples of habitual routines are as follows:

[0035] D(spend 25 minutes outside the room) → W(enter the room and walk to the table, 10 seconds) → St(stand near the chair, 2 seconds) → S(sit in the chair, 5 minutes) → W(walk to the window, 15 seconds) → St(stand near the window, 2 minutes) → W(walk to the bed, 7 seconds) → St(stand near the bed, 5 seconds) → S(sit on the bed, 90 seconds) → L(lie on the bed, 7 hours.) As the installed device monitors the user's daily behavior and habits, the system can accumulate a significant number of different habitual routines, which can be geometrically mapped as clusters of points in a multidimensional space of objects, time intervals and other parameters to represent complex user behavior. A new routine is then compared to the accumulated clusters, and if the new routine deviates significantly from each of the existing clusters, a signal can be sent to the caregiver, who can classify the new routine as the emergence of a healthy habit or a first case of deviation from healthy behavior that may require action by the caregiver. The space of habitual routines can be sent to the cloud and compared with the spaces of other users to further classify behavior and improve detection of risky situations.

[0036] Dangerous situations can be detected by the tracking device using a variety of methods:

[0037] Fall detection can be a regular feature of the state detection engine using LSTM classifiers or direct geometric processing of the radar-generated point cloud as described elsewhere herein (see, for example, the above description of the related transition steps). If a fall is detected, the system can further compare snippets of the audio recording to recordings of other transition steps to confirm the fall. The system then uses the tracking device microphone to interact with the user and contact a caregiver on the user's behalf or automatically contact a caregiver if the user does not respond.

[0038] Irregular or unexplained behaviors, for example, long periods of circulating around a room, repeatedly moving back and forth between two objects, or any significant deviation from a habitual routine, can be detected using the habitual routine detection engine described above. When such behavior is detected, it is immediately communicated to the caregiver via an alarm or alert. Attempts to communicate with the user via the tracking device's voice system are also an option.

[0039] Sleep deprivation can be detected by monitoring vital signs such as breathing, heart rate and the user's position when lying down for long periods of time.

[0040] Unwanted changes in the user's gait, such as shuffling, unsteady walking, or frequent stopping.

[0041] As mentioned above, other significant deviations from health status may be relied upon by monitoring vital signs taken periodically during stationary user conditions.

[0042] During the initial phase of functioning of the newly installed tracking device, the system builds a portrait of the room and detects various objects such as beds, tables, chairs, bookshelves, doors, windows, etc. by monitoring the absolute coordinates of the bounding boxes of the user point cloud for various user states. For example, the area of ​​the room where the walking user is habitually passing through is subtracted from the space occupied by objects. The bounding boxes of the adjacent states of standing, sitting, and lying (corresponding to the height above the floor, the height of the bed) can indicate the approximate location of the bed or couch. The location and boundaries of the bed or couch can be further enhanced by approaching the object before sitting, monitoring the positions of the bounding boxes of the point clouds of the sitting and lying states, and inverting these states after getting up. Similarly, standing for a long time near a distant point that is in the previously detected room boundary (using the subtracted walking area) implies a window.

[0043] The system embodiments described herein are explained in more detail with reference to the figures, which are briefly described below. [Brief description of the drawings]

[0044] [Figure 1] FIG. 2 is a schematic diagram of a point cloud of a room, furniture, tracking devices, and various user states according to one embodiment of the system described herein. [Diagram 2] 1 is a schematic diagram of a tracking device according to one embodiment of the system described herein. [Diagram 3] FIG. 2 is a schematic diagram of a user state and a neural network-based classifier according to one embodiment of the system described herein. [Figure 4] FIG. 2 is a schematic diagram of a user state graph according to one embodiment of the system described herein. [Diagram 5] FIG. 1 is a schematic diagram of the geometry of the tumbling process according to one embodiment of the system described herein. [Figure 6] FIG. 2 is a schematic diagram of habitual routines and associated clusters according to one embodiment of the system described herein. [Figure 7] 7A and 7B are schematic diagrams of a normal user routine and an alert user routine, respectively, according to one embodiment of the system described herein. [Figure 8] FIG. 2 is a schematic diagram of a system dashboard according to one embodiment of the system described herein. [Figure 9] FIG. 2 is a system flow diagram illustrating the system functioning in connection with identifying and classifying user states and routines, according to one embodiment of the system described herein. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0045] The system described here provides a mechanism for continuous, non-invasive, and comprehensive monitoring of older adults in long-term care facilities or other locations using ultra-wideband radar-based Internet-enabled tracking devices and novel AI-powered geometric methods of processing point clouds to detect user status, analyze user behavior, identify adverse states and conditions, and alert caregivers as needed.

[0046] Figure 1 is a schematic diagram 100 of a room 110, furniture, a tracking device, and point clouds of various user states. A long-term care recipient resides in the room 110 with a tracking device 120 plugged into an AC wall outlet. The room 110 has a door 130a, a window 130b, and includes a bed 140a, a table 140b, two chairs 140c, 140d, and a bookshelf 140e. Figure 1 shows one dynamic user state, a walking state 150, 150a, with a denser point cloud and a larger bounding box size, and three stationary user states:

[0047] · Standing near bookshelf 140e - the large box 160a represents a larger part of the user's body and will not generate a radar point due to its stationary position, however the chest 160 will generate a point due to the user's breathing.

[0048] · Sitting in chair 140c - only chest 170 and the actual point cloud and bounding box of chest 170 are shown.

[0049] Lying in bed 140a - similar to standing, only the chest 180 generates points within the body 180a.

[0050] FIG. 2 is a schematic diagram 200 of the tracking device 120. The chipset 210 allows for data collection, processing, and data exchange with the cloud. The ultra-wideband radar 220 has a coverage area close to 180 degrees and tracks objects moving within a room. If less expensive radars with smaller capture angles are utilized, the tracking device 120 may include several such less advanced radars with corresponding capture areas rotating relative to each other about a vertical axis to provide a larger combined capture area. The audio recorder 230 is used to capture sounds of daily life and user routine activities and potentially compare the daily life and user routine activities with anomalous soundtracks recorded during alarm activities. A microphone or other audio generating device 240 can be used to interact with the user (e.g., confirm a fall and provide basic instructions) or communicate with a caregiver. The power plug 250 is self-explanatory. A cellular, Wi-Fi, or other Internet connection 260 can be used to exchange data between the tracking device 120 and the cloud 270, where part of the system software runs on a server 280.

[0051] 3 is a schematic diagram 300 of user states and a neural network based classifier. The user states are described elsewhere herein, and walking 310, standing 320, sitting 330, lying (on a bed, sofa, etc.) 340, rolling over 350, and falling 360 are associated with characteristic point densities, sizes, orientations, and configurations of bounding boxes of the point cloud, indicated mnemonically by items 315, 325, 335, 345, 355, and 365. The parametric representations of the characteristic bounding boxes, point densities, and centroid locations constitute the training material for the neural network classifier. Thus, similar to the point cloud bounding box illustration of Figure 1, real-life and early radar captured point cloud samples normalized within bounding boxes 150b, 160b, 170b, 180b, 357, 367 associated with relevant user states 310a, 320a, 330a, 340a, 350a, 360a are fed into a machine learning subsystem 370, such as a recurrent neural network LSTM model. The resulting classifier 380 can then be the key system component for the continuous identification of user states.

[0052] 4 is a schematic diagram 400 of a user state graph. The various user states 310b, 320b, 330b, 340b, 350b, 360b, 410 represent the vertices of the state graph. The directed edges of the graph are represented by bold arrows and indicate the possible transitions between the states. In some user states, transitions may occur in both directions, e.g., a walking state 310b may be followed by a standing state 320b and vice versa. At the same time, a walking state 310b may be followed by a falling state 360b, but the reverse transition is not possible (at least, the user must stand up and then walk, which corresponds to a bidirectional edge between the falling state 360b and the standing state 320b).

[0053] Each possible transition between user states is associated with a transition procedure. There are a total of 16 transition procedures, such as decelerating when the walking state 310b transitions to the standing state 320b, or bending forward or backward when the walking state 310b transitions to the falling state 360b. Fourteen of these transition procedures are associated with seven pairs of vertices where the transition is available in both directions. That is, the transition modules 415, 435, 445, 455, 465, 475, 485 each contain two transitions, while two transitions 425, 495 can only be used in one direction.

[0054] Machine learning can include learning the characteristics and parameters of bounding boxes of captured point clouds for transition steps and recognizing new states by checking the transition steps that lead to the new state. Alternatively, the transition steps can be verified by direct geometric computation, for example using back-tracing of recorded user trajectories that lead to a particular condition.

[0055] Two of the user states 310b, 350b in Figure 4 are dynamic, and three states 320b, 330b, 340b are stationary, with one state 410 indicating absence of user and one state 360b being dangerous and may be accompanied by an alarm or warning 430 as described elsewhere herein. The dashed arrows connecting the stationary states with the vital signs measurement module 420 indicate optimization of the capture of vital signs such as heart rate 420a and respiration rate 420b (as described elsewhere herein).

[0056] FIG. 5 is a schematic diagram 500 of the geometry of the falling process. In the basic geometric model described herein, the user's position is represented by a bounding box in the point cloud captured by the radar of the tracking device, and the system recognizes bounding boxes that are parallel to the three-dimensional coordinate axes. Thus, if the user starts in the walking state 150c and turns, for example, forward, the intermediate position of the user's body may be represented by tilted boxes 510, 530. However, the system only recognizes standard bounding boxes and replaces the tilted boxes 510, 530 with standard bounding boxes 520, 540, which have gradually lower positions of the center of gravity and vertical height, transitioning to the final state 367a. The geometric features may be directly captured by the user trajectory processing software or may be taken into account by an automatic classifier, as described in connection with FIG. 4.

[0057] Figure 6 is a schematic diagram 600 of habitual routines and associated clusters in a multi-dimensional space of objects and time intervals. Continuous long-term monitoring of daily user behavior and habits may reveal a set of habitual routines that the user follows. A routine can be defined as a frequently repeated sequence of states, associated objects, time intervals spent in each state, and other associated parameters. Figure 6 shows two examples of habitual routines.

[0058] The first routine 610(R1) includes six user states: walking state 310c, standing near window 130b' state 320c, walking from window to bed 140a' state 310d, standing near bed state 320c, sitting in bed for a short time state 330c, and lying in bed for a long time state 340c. The second routine 620(R2) is started at the end of the first routine and includes four user states: sitting in bed 140a' state 330d (after lying in bed for a long time), standing up state 320d, walking out of the room through door 130a' state 310e, and remaining outside the room for a long time (exit state 410a).

[0059] Both sequences of user states forming routines 610, 620 can be repeated many times each day. The time interval 630 for staying in each state obviously varies. The objects encountered by the user in each routine may also be different. Thus, the bed may remain the end point of the first routine (and the start point of the second routine), but the window 130b' may be replaced, for example, by a bookshelf 140e (see FIG. 1). Different instances of a habitual routine can be interpreted as points in a multidimensional space 640, with coordinates representing, for example, objects in a room and a time sequence reflecting the stay time of each state in the routine. Clusters of points 650, 660 represent habitual routines. Thus, new routines detected by the system are similarly shown as points 670 in the same space. If a point is far away from an existing cluster, it can be automatically marked as a new routine or sequence of conditions that should be warned about by the caregiver or user, which needs to be explained.

[0060] 7A-7B are schematic diagrams of normal and warning user routines captured by the system.

[0061] Figure 7A shows a typical routine that combines two possible independent habitual routines, one of which is described in connection with Figure 6. The routine involves entering a room, walking to a chair, standing, and sitting in the chair (state 410b to state 310f to state 320e to state 330e), then standing up, walking to a bookshelf and standing near the bookshelf (state 320f to state 310g to state 320g), walking to a window and standing near the window, then walking to a bed, standing, sitting, and lying down (state 310h to state 320h to state 310i to state 320i to state 330f to state 340d, which corresponds to habitual routine 620 of Figure 6). For the reasonable time intervals occupied by each user state in the routine, it is quite normal and would not raise any suspicion of a situation that should alert the system.

[0062] 7B shows a different example of a user routine, where the user walks between the chair, the bookshelf, and the window, changing states 310j, 310k, 310l, but without stopping or sitting down, and randomly changing the walking direction (as indicated by the dashed arrows). Such restless behavior may indicate discomfort or anxiety of the user, and may cause the system to generate an alarm 710 and notify a caregiver. The system may also try to communicate with the user via the audio system (microphone) to clarify the user's actions.

[0063] FIG. 8 is a schematic diagram 800 of a system dashboard that may be displayed to a caregiver and possibly a user 810. The dashboard may include personal data 820 of the user 810, a selection of a time frame for calculating user statistics 830, daily sleep averages 840 for the selected time frame, average walking statistics 850, vital signs statistics 860 such as heart rate 860a and respiration rate 860b measured in awake and asleep states, average number of visitors and length of stay 870. Such an electronic dashboard may include a dialogue panel 880 for communication with the care center. Examples of messages may include interactions between nurses and visitors 880a, 880b entered via a message field 880d, current user status statistics (entered automatically) and alarms 880c, etc.

[0064] Referring to FIG. 9, a system flow diagram 900 illustrates a system that functions in connection with identifying and classifying user states and routines. Processing begins at step 910, where one or more tracking devices are installed in a long-term care facility, such as in a user room. After step 910, processing proceeds to step 915, where training data is collected in the form of a point cloud captured by a radar included in the tracking device. After step 915, processing proceeds to step 920, where a first round of training is performed as described elsewhere herein (see, e.g., FIG. 3 and accompanying text). After step 920, processing proceeds to step 922, where a master user geometric signature is constructed for all user states in order to identify users when there are multiple users in the room or when the master user may be temporarily absent from the room. After step 922, processing proceeds to step 925, where a state classifier is initially constructed, e.g., as described in FIG. 3 and accompanying text. After step 925, processing continues to step 930, where the system detects stationary objects in the room, such as furniture, based on the user trajectory and the furniture's complementary positions relative to the user states. After step 930, processing continues to step 932, where the system monitors the sequence of detected user states and builds initial routines, routine clusters and statistics (see in particular FIG. 6 and accompanying text).

[0065] After step 932, processing proceeds to step 935, where the system and tracking device continuously monitor the room or other facility. After step 935, processing proceeds to test step 940, where it is determined whether a point cloud has been detected by the tracking device. If not, processing proceeds to step 945, where an exit user state is set or confirmed by the system (see Figures 3, 4, 6, 7 and the accompanying text for further explanation). After step 945, processing returns to monitoring step 935, which can be reached independently from step 932. If test step 940 determines that a point cloud has been detected, processing proceeds to test step 950, where it is determined whether the point cloud corresponds to a single object. If not, processing proceeds to test step 955, where it is determined whether the guest mode of the system is on (in guest mode, the tracking device suspends tracking details of the master user until only the master user is found in the room). If so, processing returns to monitoring step 935, which can be reached independently from steps 932 and 945. If not, processing continues to step 957 where the system identifies the master user's point cloud, as described elsewhere herein.

[0066] After step 957, processing continues to test step 965, described below. If it is determined in text step 950 that the point cloud corresponds to a single object, processing continues to test step 960, which determines whether the detected object is the master user. If so, processing continues to test step 965, which can be reached independently from step 957. If not, processing returns to monitoring step 935, which can be reached independently from steps 932, 945 and test step 955 (the system monitors only the master user and ignores other individuals who may appear alone in the room, e.g., a caregiver or service person who entered the room when the master user left the room for breakfast). In test step 965, it is determined whether a new state of the master user has been detected. If not, processing returns to monitoring step 935, which can be reached independently from steps 932, 945 and test steps 955, 960. If not, processing proceeds to step 970, where the system can optionally verify the new state by geometrically back-tracing the transition steps from the previous user state to the new user state, as described elsewhere in this specification (see in particular FIG. 4 and accompanying text).

[0067] After step 970, processing proceeds to test step 972, which determines whether the new user state is a fall. If so, processing proceeds to step 985, which determines whether the system generates and initiates alarms, alerts, and notifications (including possible voice communication with the user, as described elsewhere herein). After step 985, processing is complete. If test step 972 determines that the current user state is not a fall, processing proceeds to step 975, which determines whether the new user state is added to the current user routine (e.g., the sequence of user states described in Figures 6, 7, and the accompanying text). After step 975, processing proceeds to test step 980, which determines whether the current routine is known to the system (previously recorded as a complete or partial routine). If so, processing proceeds to test step 982, which determines whether the current routine is complete (e.g., previously recorded as a habitual routine, as described elsewhere herein). If not, processing returns to monitoring step 935, which can be reached independently from steps 932, 945 and test steps 955, 960, 965. Otherwise, processing proceeds to step 995, where the statistics and cluster representation of the current routine in the multidimensional space (see FIG. 6 and accompanying text for details) are updated. After step 995, processing is complete.

[0068] If test step 980 determines that the current routine is unknown (not previously recorded), processing continues to test step 987, which determines whether the current routine is one to be warned about, e.g., as described in connection with FIG. 7B. If so, processing continues to step 985, which can be reached independently from test step 972. If not, processing continues to test step 990, which determines whether the (new, previously unknown) routine appears to be completed (or is declared complete according to system criteria). If not, processing returns to monitoring step 935, which can be reached independently from steps 932, 945 and test steps 955, 960, 965, 982. If not, processing continues to step 992, where the new routine is added to the system and associated clusters and statistics in the multidimensional space are initiated. After step 992, processing is complete.

[0069] The various embodiments described herein may be combined with each other in any suitable combination in connection with the systems described herein. Additionally, in some cases, the order of steps of the flowcharts, flow diagrams, and / or described flow processes may be changed as necessary. The configuration and functionality of the system may subsequently differ from the diagrams shown herein. Additionally, various aspects of the systems described herein may be implemented using various applications and deployed on various devices, including, but not limited to, smartphones, tablets, and other mobile computers. The smartphones and tablets may use operating systems selected from the group consisting of iOS, Android OS, Windows Phone OS, Blackberry OS, and mobile versions of Linux OS. The mobile computers and tablets may use operating systems selected from the group consisting of Mac OS, Windows OS, Linux OS, and Chrome OS.

[0070] Software implementations of the systems described herein may include executable code stored on a computer-readable medium and executed by one or more processors. The computer-readable medium may be non-transitory and may include a computer hard drive, a ROM, a RAM, a flash memory, a portable computer storage medium such as a CD-ROM, a DVD-ROM, a flash drive, an SD card, and / or other drives with, for example, a Universal Serial Bus (USB) interface, and / or other suitable tangible or non-transitory computer-readable medium or computer memory on which executable code is stored and executed by a processor. The software may be bundled (pre-loaded), installed from an app store, or downloaded from a network operator location. The systems described herein may be used in connection with any suitable operating system.

[0071] Other embodiments of the invention will be apparent to those skilled in the art from consideration of the specification or practice of the invention disclosed herein. It is intended that the specification and examples be considered as exemplary only, with a true scope and spirit of the invention being indicated by the following claims.

Claims

1. 1. A method for determining a physical condition of a person, comprising: detecting the positions of different parts of the person in a given space; converting the detected positions of different parts of the person in a given space into a point cloud having a point density that varies according to the movement of each of the different parts; correlating the motion and spatial position data from the point cloud with locations of known physical states and transitions between different states; selecting a particular physical state including a location in the predetermined space by matching the data from the point cloud to the particular physical state; The method includes:

2. The method of claim 1 , wherein the positions of different parts of the person are detected using a tracking device.

3. The method of claim 2 , wherein the tracking device is a contactless tracking device.

4. The method of claim 2 , wherein the tracking device includes at least one wideband radar.

5. The method of claim 4 , wherein the tracking device communicates wirelessly with at least one server in a cloud computing system.

6. The method of claim 1 , wherein the state further comprises at least one of a walking state, a standing state, a sitting state, a lying state, a rolling state, a falling state, and an exiting state.

7. The method of claim 6 , wherein a fall is detected in response to the person transitioning from the walking state to the lying state.

8. The method of claim 7 , wherein during the transition, the person is detected at a lower and lower center of gravity of the point cloud.

9. The method of claim 7 , wherein a voice generating device communicates with the person following the fall to confirm the fall and provide basic instructions to the person.

10. 10. The method of claim 9, further comprising contacting a caregiver if the person does not move or interact following the voice generating device communicating with the person.

11. The method of claim 10 , wherein the audio generating device includes a microphone for receiving audio communication from the person.

12. The method of claim 1 , wherein each of the states is associated with a point density, a size, an orientation, a center of gravity, and a bounding box placement of the point cloud.

13. The method of claim 12 , in which parametric representations of the bounding boxes, the point densities, and the centroid locations of samples of different states are provided as inputs to a neural network classifier.

14. 14. The method of claim 13, wherein the neural network is trained by providing the neural network to a server of the cloud computing system that detects the positions of different parts of the person in a detected predetermined space and receives data from a tracking device that is in wireless communication with the cloud computing system.

15. The method of claim 14, wherein the neural network is a long short-term memory recurrent neural network.

16. The method of claim 13 , wherein the neural network classifier correlates motion and position data from the point cloud with known physical state locations and transitions between different states to select the particular physical state.

17. The method of claim 1 , further comprising maintaining information corresponding to habitual routine state transitions and positions of the person.

18. 20. The method of claim 17, wherein habitual routine state transitions and locations are determined by detecting clusters of points in a multi-dimensional space of objects, time intervals, locations, and sequences of state transitions that represent complex user behavior.

19. 20. The method of claim 17, wherein an alert is provided to a caregiver in response to the person deviating from the habitual routine state transition and position.

20. 20. The method of claim 19, wherein deviating from the habitual routine state transitions and positions includes cycling around a room for an extended period of time or moving back and forth repeatedly between two objects.

21. 20. The method of claim 18, wherein the clusters of points corresponding to the habitual routines are provided to a cloud computing system for comparison with clusters of points corresponding to other people's habitual routines to further classify the behavior and improve detection of risky situations.

22. The predetermined space is a room, The method of claim 1 , wherein the person is in the room and objects within the room are initially detected by monitoring absolute coordinates of bounding boxes of the point cloud at various user states.

23. 23. The method of claim 22, wherein the object comprises at least one of a bed, a table, a chair, a bookshelf, a door, or a window.

24. The method of claim 22 , wherein objects within the room are detected by subtracting multiple positions of the bounding box from the area of ​​the room.

25. 25. The method of claim 24, wherein a bed is detected by observing that the person is in a lying position at a particular height above the floor.

26. 26. The method of claim 25, wherein the boundaries of the bed are determined by tracking bounding boxes corresponding to a lying position, a sitting position before or after lying position, and a standing position before sitting.

27. 25. The method of claim 24, wherein the bed or couch is detected by observing adjacent positions of standing, sitting, and lying in positions corresponding to the bed or couch.

28. 25. The method of claim 24, wherein windows are detected by observing that the person stands at a boundary of the room for a relatively long period of time.

29. 1. A non-transitory computer readable medium storing software for determining a physical state of a person, the software comprising: executable code for detecting the position of different parts of the person in a given space; executable code for converting the detected positions of different parts of the person in a predetermined space into a point cloud having a point density that varies according to the movement of each of the different parts; executable code for correlating the motion and predetermined position data from the point cloud with positions of known physical states and transitions between different states; executable code for selecting a particular physical state including a location in the predetermined space by matching the data from the point cloud to the particular physical state; A non-transitory computer readable medium comprising:

Images