System and method of falling detection

Abstract

A falling detection system includes a sensor, a target tracking module, a time series feature extraction module, a falling action classifier, a target finite state machine module, an object recognition classifier and an alarm module. The sensor is utilized to obtain all point cloud data in the sensing environment. The target tracking module is utilized to identify the target point cloud data from all point cloud data. The time series feature extraction module is utilized to process the target point cloud data utilizing lightweight time series target feature extraction technology and output a time series feature matrix. The falling action classifier is utilized to output a falling action classification result. The target finite state machine module is utilized to calculate state duration of a target finite state machine corresponding to a target object. The object recognition classifier is utilized to output an object recognition result. The alarm module is utilized to output a warning message based on the state duration and the object recognition result.

Description

Fall detection system and fall detection method This invention relates to a detection system and a detection method, and more particularly to a fall detection system and a fall detection method. Conventional radar-based fall detection technology is primarily based on deep learning methods. However, most deep learning methods rely solely on the output of a fall action classifier as the basis for fall alerts, failing to differentiate between stages of the fall behavior or utilize finite state machines to track the fall process and the object's state after landing. Therefore, fall detection systems implemented with existing technology are prone to false alarms due to single misclassifications, thus reducing detection accuracy. Furthermore, existing fall detection systems also cannot determine whether the object gets up after falling, nor do they calculate the duration of the fall, making it difficult to accurately grasp the object's real-time condition. The purpose of this invention is to provide a fall detection system and a fall detection method. The system uses lightweight temporal target feature extraction technology to process point cloud data of a person under test and outputs a temporal feature matrix. A fall action classifier based on a deep learning model then processes the temporal feature matrix and outputs a fall action classification result to determine the current state of the person under test and calculate the state dwell time of the corresponding target finite state machine. Next, an object recognition classifier, also based on a deep learning model, processes the temporal feature matrix and outputs an object recognition result. Finally, the state dwell time of the target finite state machine and the object recognition result output by the object recognition classifier are used as the basis for whether the fall detection system displays a warning message. One aspect of this invention provides a fall detection system, comprising a sensor, a target tracking module, a temporal feature extraction module, a fall action classifier, a target finite-state machine module, an object recognition classifier, and an alarm module. The sensor is configured to acquire at least one point cloud of data in the sensing environment. The target tracking module is configured to identify at least one target point cloud of a target object from the point cloud data. The temporal feature extraction module is configured to process the target point cloud data using lightweight time-series target feature extraction technology and output a temporal feature matrix. The fall action classifier is configured to receive the temporal feature matrix and output a fall action classification result. The target finite-state machine module is configured to receive the fall action classification result and calculate the state dwell time of the target finite-state machine corresponding to the target object. The object recognition classifier is configured to receive the temporal feature matrix and output an object recognition result. The alarm module is configured to issue an alarm message based on the state dwell time and the object recognition result. In some embodiments, the temporal feature extraction module is further configured to extract 12-dimensional temporal series features of the target object in three-dimensional space from target point cloud information using lightweight temporal target feature extraction technology. In some embodiments, the 12-dimensional time series features include at least one of location, distance, angle, signal-to-noise ratio (SNR), and velocity. In some embodiments, the time series feature matrix is ​​a 12-dimensional time series feature matrix, and the time series feature extraction module is further configured to generate and compress the 12-dimensional time series features into a 12-dimensional time series feature matrix. In some embodiments, the fall action classifier is based on a deep learning model. In some embodiments, the target finite state machine module is further configured to define three states of the target finite state machine, namely, normal activity state, falling state and falling state, and calculate the state dwell time of the target finite state machine of the corresponding target object based on the normal activity state, falling state and falling state. In some embodiments, the target finite state machine module is further configured to calculate the target get-up height of the target object using an adaptive get-up height threshold algorithm. In some embodiments, the fall action classification result is a normal activity action or a falling action, and the object identification result is a person falling or a non-person falling. In some embodiments, when the state dwell time is greater than a preset time threshold and the object recognition result is a person falling, the alarm module issues a warning message. Another aspect of the present invention provides a fall detection method, comprising the following steps: acquiring at least one point cloud data in a sensing environment by a sensor; identifying at least one target point cloud data of a target object from the point cloud data by a target tracking module; processing the target point cloud data using a lightweight temporal target feature extraction technology by a temporal feature extraction module and outputting a temporal feature matrix; receiving the temporal feature matrix by a fall action classifier and outputting a fall action classification result; receiving the fall action classification result by a target finite state machine module and calculating the state dwell time of the target finite state machine corresponding to the target object; receiving the temporal feature matrix by an object recognition classifier and outputting an object recognition result; and issuing a warning message by an alarm module based on the state dwell time and the object recognition result. The embodiments of the present invention will be discussed in detail below. It will be understood that the embodiments provide many applicable concepts that can be implemented in a wide variety of specific contexts. The discussed and disclosed embodiments are for illustrative purposes only and are not intended to limit the scope of the invention.Figure 1 is a functional block diagram of a fall detection system 100 according to an embodiment of the present invention. The fall detection system 100 includes a sensor 110, a target tracking module 120, a temporal feature extraction module 130, a fall action classifier 140, a target finite-state machine module 150, an object recognition classifier 160, and an alarm module 170. The sensor 110 is configured to acquire at least one point cloud of data in the sensing environment. The target tracking module 120 is configured to identify at least one target point cloud of a target object (i.e., the person being tested by the fall detection system 100) from the point cloud data. The temporal feature extraction module 130 is configured to process the target point cloud data using lightweight time series target feature extraction technology and output a temporal feature matrix. The fall action classifier 140 is configured to receive the temporal feature matrix and output a fall action classification result. The target finite state machine module 150 is configured to receive the fall action classification results and calculate the state dwell time of the corresponding target object's finite state machine. The object recognition classifier 160 is configured to receive the temporal feature matrix and output the object recognition results. The alarm module 170 is configured to issue an alarm message based on the state dwell time and the object recognition results.In one embodiment of the present invention, the fall detection system 100 further includes a memory and a processor (not shown in FIG1). The memory is configured to store one or more instructions for implementing the operation of multiple modules (including but not limited to the sensor 110, the target tracking module 120, the timing feature extraction module 130, the fall action classifier 140, the target finite state machine module 150, the object recognition classifier 160, and the alarm module 170). The memory may be random access memory (RAM), read-only memory (ROM), flash memory, solid state drive (SSD), or other similar elements or combinations thereof, but is not limited thereto. The processor is configured to execute the instructions stored in the memory to complete the functions of each module. The processor may be a central processing unit (CPU), graphics processing unit (GPU), microcontroller unit (MCU), microprocessor, system-on-chip (SoC), or digital signal processor. The processor (DSP), application-specific integrated circuit (ASIC), programmable logic controller (PLC), or a combination of the above, but not limited to these. In one embodiment of the present invention, the sensor 110 in the fall detection system 100 may be a radar to collect point cloud data in the sensing environment. However, the present invention is not limited to using radar to achieve the function of point cloud data collection. Furthermore, the radar used in the fall detection system 100 may be a millimeter wave radar, which is a radar that operates in the millimeter wave band, transmits millimeter waves through an antenna, and receives signals reflected back from the target / obstacle to calculate the relative speed, distance, and angle with the target. After receiving all point cloud data of the sensing environment obtained by sensor 110 (i.e., millimeter-wave radar), target tracking module 120 then identifies the target point cloud data of the target object from all the point cloud data. Specifically, target tracking module 120 processes all the collected radar point cloud data through a clustering algorithm and determines the target point cloud data belonging to the target object. After receiving the target point cloud data output by the target tracking module 120, the temporal feature extraction module 130 processes the target point cloud data using lightweight temporal target feature extraction technology and outputs a temporal feature matrix. Specifically, the lightweight temporal target feature extraction technology performs estimation processes such as calculating the summation average, calculating the standard deviation, and taking the maximum and minimum values ​​from the target point cloud data to assign the target object 12 features, including position (i.e., X coordinates, Y coordinates, Z coordinates), signal-to-noise ratio (SNR), distance, horizontal angle, pitch angle, radial velocity, radial velocity standard deviation, maximum radial velocity, minimum radial velocity, and maximum and minimum radial velocity difference. That is, it extracts 12-dimensional time-series features of the target object in three-dimensional space from the target point cloud information. Then, these extracted features are passed into a buffer that can store 25 frames of target object features to form a 25-column, 12-column 12-dimensional temporal feature matrix. Finally, the summation average is calculated in the time domain in units of a specific number of frames to achieve the effect of compressing the temporal feature matrix. In one embodiment of the present invention, if the average is calculated in units of 5 frames in the time domain, a lightweight 12-dimensional temporal feature matrix with 5 columns and 12 columns will be output. This improves the classification efficiency of the classifier used in subsequent processes and is suitable for computation on a microcomputer. It should also be noted that this embodiment only uses the calculation of the average in units of 5 frames in the time domain as an example. However, the present invention does not limit the number of frames used. That is, in other embodiments of the present invention, if the average is calculated in units of other frames in the time domain, a lightweight 12-dimensional temporal feature matrix with other numbers of columns and 12 columns will be output, achieving the same effect of compressing the temporal feature matrix. After receiving the lightweight temporal feature matrix output by the temporal feature extraction module 130, the fall action classifier 140 then outputs the fall action classification result. Specifically, the fall action classifier 140 is built based on a deep learning model. In one embodiment of the present invention, the fall action classifier 140 may be built based on, for example, a long short-term memory (LSTM) deep learning model. This LSTM deep learning model architecture includes a 10-unit LSTM layer and a dropout layer, used to classify the temporal feature matrix of the input model to determine whether the fall action classification result of the target object is "normal activity action" or "falling action". This fall action classification result of the target object is the output of the fall action classifier 140 built based on the deep learning model. It is worth mentioning that the fall action classification result output by the fall action classifier 140 is then received and processed by the target finite state machine module 150. An alarm is not immediately triggered when the fall action classification result of the target object is classified as a falling action, thus significantly reducing the probability of false alarms. It should also be noted that this embodiment only uses a long short-term memory deep learning model architecture to build the fall action classifier 140 as an example. However, this invention does not limit the deep learning model architecture used. That is, in other embodiments of this invention, the fall action classifier 140 can be implemented using other deep learning model architectures different from the long short-term memory deep learning model architecture. After receiving the fall action classification result output by the fall action classifier 140, the target finite state machine module 150 then calculates the state dwell time of the target object's finite state machine. Specifically, the complete fall action of the target object can be roughly divided into three stages: the initial stabilization stage, the falling stage, and the landing stage. These three stages can correspond to the three states of the target object's finite state machine: normal activity state, falling state, and landing state, respectively. Refer to the schematic diagram of the target finite state machine shown in Figure 2. In the initial stage of fall detection, the target object will directly enter the normal activity state. Then, based on the fall action classification result output by the fall action classifier 140, it is determined whether the target object will transition from the normal activity state to the falling state. Furthermore, if the fall action classifier 140 outputs a fall action classification result of "falling action", and the target object has also transitioned from a normal activity state to a falling state (as shown in path A in Figure 2), then the target finite state machine module 150 will observe whether the fall action classifier 140 continues to output a fall action classification result of "falling action" (i.e., whether the target finite state machine module 150 continues to receive fall action category frames), and set a continuous fall action category frame threshold to further confirm which state of the target finite state machine the target object is in. In one embodiment of the present invention, when the number of fall action category frames continuously received by the target finite state machine module 150 is less than the set threshold for continuous fall action category frames, and the next frame received is displayed as a non-fall action category frame, the target finite state machine module 150 will return the state of the target object to the normal activity state, as shown in path B in FIG2. In one embodiment of the present invention, when the number of consecutive fall action category frames received by the target finite state machine module 150 is greater than or equal to the set threshold for consecutive fall action category frames, and the next frame received is displayed as a non-fall action category frame, the target finite state machine module 150 will transfer the state of the target object from the falling state to the fallen state, as shown by path C in FIG2. In one embodiment of the present invention, when the target object transitions from a falling state to a fallen state, the target finite state machine module 150 calculates the target get-up height of the target object using an adaptive get-up height threshold algorithm. Specifically, the adaptive get-up height threshold algorithm sets the get-up height threshold by adding the initial fall height and the final fall height of the target object and taking the average, supplemented by a preset get-up height constant. In one embodiment of the present invention, if the target get-up height of the frame currently received by the target finite state machine module 150 is less than this get-up height threshold, the target object remains in the fallen state; on the other hand, if the target get-up height of the frame currently received by the target finite state machine module 150 is greater than or equal to this get-up height threshold, the number of get-ups of the target object is incremented by one. When the number of get-ups of the target object exceeds a preset get-up count threshold, the target finite state machine module 150 transitions the target object's state from the fallen state back to the normal active state, as shown by path D in Figure 2. It should be noted that regardless of whether the target finite state machine module 150 transfers the state of the target object from the normal active state to the falling state or from the falling state to the fallen state, it will calculate the duration of the target object in the falling state or the fallen state. In other words, the target finite state machine module 150 will calculate the duration of the falling state and the duration of the fallen state separately, which will be used as a reference for whether to trigger the fall alarm. After receiving the lightweight temporal feature matrix output by the temporal feature extraction module 130, the object recognition classifier 160 then outputs the object recognition result. Specifically, the object recognition classifier 160 can also be built based on a deep learning model. In one embodiment of the present invention, the object recognition classifier 160 can be built based on, for example, a long short-term memory deep learning model. This long short-term memory deep learning model architecture includes a long short-term memory layer with four units and two batch normalization layers, used to classify the temporal feature matrix of the input model to determine whether the object recognition result of the target object is "person fell" or "non-person fell". This object recognition result of the target object is the output of the object recognition classifier 160 built based on the deep learning model. It should also be noted that this embodiment only uses the Long Short-Term Memory deep learning model architecture to build the object recognition classifier 160 as an example for illustration. However, the present invention does not limit the deep learning model architecture used. That is to say, in other embodiments of the present invention, the object recognition classifier 160 can be implemented using other deep learning model architectures different from the Long Short-Term Memory deep learning model architecture. The fall detection system 100 also includes an alarm module 170, which uses the state dwell time of the target object (i.e., the dwell time of the falling state and the dwell time of the fallen state) calculated by the target finite state machine module 150 as the basis for whether to issue an alarm message. Furthermore, the alarm module 170 also determines whether to issue an alarm message based on the object identification result (i.e., "person falls" or "non-person falls") output by the object identification classifier 160, thereby filtering out false alarms generated by non-person targets and improving the detection accuracy of the fall detection system 100. In one embodiment of the present invention, when the state dwell time calculated by the target finite state machine module 150 is greater than a preset time threshold and the object identification result output by the object identification classifier 160 is "person falls," the alarm module 170 issues an alarm message. Figure 3 is a flowchart of a fall detection method 300 according to an embodiment of the present invention. The fall detection method 300 can be implemented by, for example, the fall detection system 100 shown in Figure 1, which includes a sensor 110, a target tracking module 120, a temporal feature extraction module 130, a fall action classifier 140, a target finite state machine module 150, an object recognition classifier 160, and an alarm module 170, or other similar systems. The fall detection method 300 includes steps S310 to S370. The following paragraphs describe the implementation method of each step in the fall detection method 300 in conjunction with the content of Figures 1-3. Step S310: The sensor acquires all point cloud data in the sensing environment. Step S320: The target tracking module identifies the target point cloud data of the target object from all point cloud data. Step S330: The temporal feature extraction module processes the target point cloud data using lightweight temporal target feature extraction technology and outputs a temporal feature matrix. Step S340: The fall action classifier receives the temporal feature matrix and outputs the fall action classification result. Step S350: The target finite state machine module receives the fall action classification result and calculates the state dwell time of the target finite state machine of the corresponding target object. Step S360: The object identification classifier receives the temporal feature matrix and outputs the object identification result. Step S370: The alarm module issues a warning message based on the state dwell time and the object recognition result. For an explanation of each step S310-S370, please refer to the operation of each component in the fall detection system 100 shown in Figure 1, for example, which will not be repeated here. In summary, the fall detection system and method of this invention process the point cloud data of the target person using lightweight temporal target feature extraction technology and output a temporal feature matrix. Then, a fall action classifier based on a deep learning model processes the temporal feature matrix and outputs a fall action classification result, which is used to determine the current state of the target person and calculate the state dwell time of the target person's target finite state machine. Next, an object recognition classifier based on the same deep learning model processes the temporal feature matrix and outputs an object recognition result. Finally, the state dwell time of the target finite state machine and the object recognition result output by the object recognition classifier are used as the basis for whether the fall detection system displays a warning message. Although the present invention has been disclosed above by way of embodiments, it is not intended to limit the scope of the present invention. Anyone skilled in the art can make various changes, substitutions and modifications without departing from the spirit and scope of the present invention. Therefore, the scope of protection of the present invention shall be defined by the appended claims. 100: Fall detection system; 110: Sensor; 120: Target tracking module; 130: Temporal feature extraction module; 140: Fall action classifier; 150: Target finite state machine module; 160: Object recognition classifier; 170: Alarm module; 300: Fall detection method; S310, S320, S330, S340, S350, S360, S370: Steps A, B, C, D: Path To make the above and other objects, features, advantages and embodiments of the present invention more apparent and understandable, the accompanying drawings are described below: Figure 1 is a functional block diagram of a fall detection system according to an embodiment of the present invention; Figure 2 is a schematic diagram of a target finite state machine according to an embodiment of the present invention; and Figure 3 is a flowchart of a fall detection method according to an embodiment of the present invention. Domestic storage information (please note in order of storage institution, date, and number): None. International storage information (please note in order of storage country, institution, date, and number): None. 100: Fall Detection System 110: Sensor 120: Target Tracking Module 130: Temporal Feature Extraction Module 140: Fall Action Classifier 150: Target Finite State Machine Module 160: Object Identification and Classification 170: Alarm Module

Claims

1. A fall detection system includes: a sensor configured to acquire at least one point cloud of data in a sensing environment; a target tracking module configured to identify at least one target point cloud of a target object from the at least one point cloud of data; a temporal feature extraction module configured to process the at least one target point cloud of data using a lightweight time series target feature extraction technique and output a temporal feature matrix, wherein the temporal feature extraction module is further configured to extract a 12-dimensional time series feature of the target object in three-dimensional space from the target point cloud information using the lightweight time series target feature extraction technique; a fall action classifier configured to receive the temporal feature matrix and output a fall action classification result; a target finite-state machine module configured to receive the fall action classification result and calculate the state dwell time of the target finite-state machine corresponding to the target object; and an object recognition module. A recognition classifier is configured to receive the temporal feature matrix and output an object recognition result; and an alarm module is configured to issue an alarm message based on the state dwell time and the object recognition result. The fall detection system as described in claim 1, wherein the 12-dimensional time-series features include at least one of a position, a distance, an angle, a signal-to-noise ratio (SNR), and a velocity. The fall detection system as described in claim 1, wherein the time series feature matrix is ​​a 12-dimensional time series feature matrix, and the time series feature extraction module is further configured to generate and compress the 12-dimensional time series features into the 12-dimensional time series feature matrix. The fall detection system as described in claim 1, wherein the fall action classifier is based on a deep learning model. As described in claim 1, the fall detection system further includes a target finite state machine module configured to define three states of the target finite state machine: a normal activity state, a falling state, and a falling state, and to calculate the dwell time of the target finite state machine in the corresponding state based on the normal activity state, the falling state, and the falling state. As described in claim 5, the fall detection system wherein the target finite state machine module is further configured to calculate the target's get-up height using an adaptive get-up height threshold algorithm. The fall detection system as described in claim 1, wherein the fall action classification result is a normal activity action or a falling action, and wherein the object identification result is a person falling or a non-person falling. As described in claim 7, in the fall detection system, when the state stay time is greater than a preset time threshold and the object recognition result is that the person has fallen, the alarm module issues the warning message. A fall detection method includes: acquiring at least one point cloud data in a sensing environment by a sensor; identifying at least one target point cloud data of a target object from the at least one point cloud data by a target tracking module; processing the at least one target point cloud data by a temporal feature extraction module using a lightweight temporal target feature extraction technology and outputting a temporal feature matrix, wherein the temporal feature extraction module is further configured to extract a 12-dimensional time series feature of the target object in three-dimensional space from the target point cloud information using the lightweight temporal target feature extraction technology; receiving the temporal feature matrix by a fall action classifier and outputting a fall action classification result; receiving the fall action classification result by a target finite state machine module and calculating a state dwell time of a target finite state machine corresponding to the target object; receiving the temporal feature matrix by an object recognition classifier and outputting an object recognition result; and issuing a warning message by an alarm module based on the state dwell time and the object recognition result.

Images