Fall monitoring method, device, system and storage medium

Abstract

The embodiments of the present application relate to the field of deep learning technology and disclose a fall monitoring method, device, system, and storage medium. The method includes: collecting movement data of a target object; obtaining movement data within at least two time periods according to the collection order of the movement data; determining feature vectors based on the movement data within each time period; inputting each feature vector into a long short-term memory neural network model to obtain a first output result, wherein the first output result of the current time period is determined based on the first output result of the previous time period and the feature vector of the current time period. If the current time period is the first time period, the first output result of the previous time period uses an initialization value; inputting the first output result corresponding to each feature vector into an attention model to obtain a second output result; and judging whether the target object has fallen based on the first output result and the second output result. The embodiments of the present application achieve both accurate and rapid judgment of whether the target object has fallen.

Description

Technical Field

[0001] The embodiments of the present application relate to the field of deep learning technology, and specifically to a fall monitoring method, apparatus, device, and storage medium. Background Art

[0002] As China's aging population accelerates, the elderly population is growing, and the resulting problems of empty-nest elderly and elderly health are becoming increasingly prominent. It's well known that falls can have adverse health effects on the elderly. Furthermore, young children, who are active and lively, are particularly prone to falls, making it difficult for parents to constantly supervise them. Furthermore, young and middle-aged individuals who engage in physically or mentally demanding jobs, such as delivery drivers, those working at heights, construction workers, programmers, and white-collar workers, are also prone to falls in their daily lives.

[0003] When someone falls, if they are unconscious and unable to call for help or if no one is around to see them and they cannot receive timely treatment, it is very easy to miss the best time to rescue them and the situation will worsen. Therefore, if the target person can be monitored and judged whether they have fallen, when a fall occurs, rescue personnel can rush to the scene to provide treatment in time and reduce the damage. Summary of the Invention

[0004] In view of the above problems, the embodiments of the present application provide a fall monitoring method, device, system and storage medium to solve the problem in the prior art that there is a lack of an effective and rapid method for determining whether a target object has fallen.

[0005] According to one aspect of an embodiment of the present application, a fall monitoring method is provided, the method comprising: continuously collecting movement data of a target object; sequentially acquiring the movement data in at least two time periods according to the collection time sequence of the movement data, wherein the duration of each time period is the same; determining a feature vector for each time period based on the movement data in each time period; inputting each feature vector into a long short-term memory neural network model, and obtaining a first output result corresponding to each feature vector, wherein the first output result of a current time period is determined based on the first output result of a previous time period and the feature vector of the current time period, and if the current time period is the first time period, the first output result of the previous time period adopts an initialization value; inputting the first output result corresponding to each feature vector into an attention model, and obtaining a second output result; and judging whether the target object has fallen based on the first output result and the second output result.

[0006] In an optional embodiment, before sequentially acquiring the movement data within at least two time periods according to the collection time sequence of the movement data, the method further includes: acquiring the movement data within a preset time length; performing a pre-fall judgment on the target object based on the movement data within the preset time length to determine whether the target object has pre-fallen; if it is determined that the target object has pre-fallen, continuing to acquire at least one of the movement data; sequentially acquiring the movement data within at least two time periods according to the collection time sequence of the movement data further includes: sequentially acquiring the movement data within at least two time periods according to the collection time sequence of the movement data, wherein the acquired movement data includes at least one movement data collected after determining that the target object has pre-fallen.

[0007] In an optional manner, the pre-judgment of a fall of the target object based on the movement data within the preset time period to determine whether the target object has pre-fallen further includes: obtaining a maximum change in the movement data within the preset time period; if the maximum change is greater than a preset threshold, determining that the target object has pre-fallen.

[0008] In an optional embodiment, before determining the characteristic vector of each time period based on the movement data in each time period, the method further includes: preprocessing the movement data in each time period to remove baseline drift and / or noise to obtain preprocessed movement data; determining the characteristic vector of each time period based on the movement data in each time period further includes: determining the characteristic vector based on the preprocessed movement data in each time period.

[0009] In an optional manner, determining the characteristic vector of each time period based on the movement data in each time period further includes: determining the average value, variance, maximum value, minimum value and / or the difference between the maximum value and the minimum value in the movement data in each time period; performing a fast Fourier transform on the movement data in each time period to obtain a fast Fourier transform result; obtaining the frequency value corresponding to the maximum amplitude in the fast Fourier transform result in each time period; and constructing the characteristic vector based on the average value, variance, difference, frequency value corresponding to the maximum amplitude in each time period and / or the movement data last collected in the time period.

[0010] In an optional manner, judging whether the target object falls according to the first output result and the second output result further includes: multiplying the first output result and the second output result corresponding to each of the feature vectors to obtain the product corresponding to each of the feature vectors; adding the products corresponding to each of the feature vectors to obtain a sum; inputting the sum into a multi-layer perceptron to obtain a third output result; inputting the third output result into a normalized exponential function layer, and judging whether the target object falls according to the normalization processing result of the normalized exponential function layer.

[0011] In an optional manner, the steps of determining the characteristic vector of each time period based on the movement data within each time period and the steps preceding it are performed by a smart wearable device, and the steps of inputting each characteristic vector into a long short-term memory neural network model, obtaining the first output result corresponding to each characteristic vector and the steps following it are performed by a mobile terminal associated with the smart wearable device. After determining the characteristic vector of each time period based on the movement data within each time period, the method further includes: the smart wearable device transmitting the characteristic vector to the mobile terminal.

[0012] According to another aspect of an embodiment of the present application, a fall monitoring device is provided, comprising: a processor, a memory, a communication interface, and a communication bus, wherein the processor, the memory, and the communication interface communicate with each other via the communication bus; the memory is used to store at least one executable instruction, and the executable instruction enables the processor to perform operations such as the above-mentioned fall monitoring method.

[0013] According to another aspect of an embodiment of the present application, a fall monitoring system is provided, comprising: a smart wearable device, configured to continuously collect movement data of a target object; and configured to sequentially obtain the movement data of at least two time periods according to a time sequence of the movement data collection, wherein each of the time periods has the same duration; and further configured to determine a feature vector for each of the time periods based on the movement data within the time period;

[0014] A mobile terminal, used to input each of the feature vectors into a long short-term memory neural network model, and obtain a first output result corresponding to each of the feature vectors, wherein the first output result of the current time period is determined based on the first output result of the previous time period and the feature vector of the current time period. If the current time period is the first time period, the first output result of the previous time period adopts an initialization value; and used to input the first output result corresponding to each of the feature vectors into an attention model to obtain a second output result; and also used to determine whether the target object has fallen based on the first output result and the second output result.

[0015] According to another aspect of an embodiment of the present application, a computer-readable storage medium is provided, wherein the storage medium stores executable instructions, and the executable instructions enable a fall monitoring device to perform operations corresponding to the above-mentioned fall monitoring method.

[0016] In the embodiments of the present application, by continuously collecting the target object's motion data and constructing at least two feature vectors using the motion data from at least two time periods for analysis and judgment, the number of samples analyzed is increased compared to analyzing and judging only the current moment's motion data, thereby improving the accuracy of the analysis. Furthermore, since not all collected motion data is used for analysis and judgment, but only a portion of the collected motion data is used to construct feature vectors for analysis and judgment, the amount of data analyzed is reduced compared to analyzing all collected motion data, thereby improving the speed of analysis and judgment. Each feature vector is then input into a long-short-term memory neural network model to obtain a first output result corresponding to each vector. Because the long-short-term memory neural network model uses an attention mechanism to obtain the first output result of the current time period, that is, the first output result of the current time period is determined based on the first output result of the previous time period and the feature vector of the current time period, thereby improving the accuracy of obtaining the first output result. Each first output result is then input into the attention model to obtain the corresponding second output result. Finally, based on the first and second output results, it is determined whether the target object has fallen. Since the first output result is obtained at a faster speed and with a higher accuracy, the speed of further using the first output result to determine whether the target object has fallen is also faster, and the accuracy of the obtained determination result is also higher.

[0017] The above description is only an overview of the technical solutions of the embodiments of the present application. In order to more clearly understand the technical means of the embodiments of the present application, they can be implemented in accordance with the contents of the specification. In order to make the above and other purposes, features and advantages of the embodiments of the present application more obvious and easy to understand, the specific implementation methods of the present application are listed below. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] The accompanying drawings are only used to illustrate the embodiments and are not to be considered as limiting the present application. In addition, the same reference symbols are used to represent the same components throughout the drawings. In the drawings:

[0019] Figure 1 A schematic diagram of the process of a fall monitoring method provided in an embodiment of the present application is shown;

[0020] Figure 2 A schematic diagram showing a flow chart of a fall monitoring method provided by another embodiment of the present application is shown;

[0021] Figure 3 A flow chart of a fall monitoring method provided in another embodiment of the present application is shown;

[0022] Figure 4 Shown Figure 1 Schematic diagram of the sub-step flow chart of step 130;

[0023] Figure 5 Shown Figure 1 A schematic flow chart of the sub-steps of step 160;

[0024] Figure 6 A schematic structural diagram of a fall monitoring device provided in an embodiment of the present application is shown;

[0025] Figure 7 A structural diagram of a fall monitoring system provided in an embodiment of the present application is shown. DETAILED DESCRIPTION

[0026] The exemplary embodiments of the present application will be described in more detail below with reference to the accompanying drawings. Although exemplary embodiments of the present application are shown in the drawings, it should be understood that the present application can be implemented in various forms and should not be limited to the embodiments set forth herein.

[0027] When a target person falls seriously, if emergency personnel can arrive at the scene in time to provide treatment, the damage caused by the fall can be reduced. Therefore, it is necessary to monitor the target person and quickly and effectively determine whether a fall has occurred, so as to prevent the fall from not being discovered in time and missing the best time for treatment.

[0028] The inventors of this application have discovered that some prior art methods typically use only the current moment's movement data to determine whether a fall has occurred. This analysis of only the current moment's movement data results in a small number of data samples, and the failure to analyze movement data at different moments can easily lead to problems such as low accuracy in the judgment results. However, in other prior art methods, in order to improve the accuracy of the judgment results, the movement data of the target person during their movement is collected in real time, and all collected movement data is analyzed to determine whether a fall has occurred. However, due to the large amount of data analyzed, this can easily lead to problems such as low judgment efficiency, resulting in missing the golden rescue time.

[0029] Based on the above considerations, in order to accurately and quickly determine whether the target person has fallen, the inventors of this application have conducted in-depth research and proposed a fall monitoring method. This method continuously collects the movement data of the target object, obtains the movement data of at least two time periods according to the collection time sequence of the movement data, and then uses the movement data in each time period to determine the feature vector, inputs each feature vector into the long short-term memory neural network model, and then inputs the output results of the long short-term memory neural network model into the attention model, thereby determining whether the target object has fallen based on the output results of the long short-term memory neural network model and the output results of the attention model. This method constructs the feature vectors in different time periods as the input of the long short-term memory neural network model, which not only utilizes the movement data collected at different times, but also limits the input amount of the long short-term memory neural network model and the attention model, so that it can accurately and quickly determine whether the target object has fallen.

[0030] The fall monitoring method disclosed in the embodiment of the present application can be used for, but is not limited to, monitoring all movable objects such as humans, animals, robots, etc. The embodiment of the present application is only described by using the example of monitoring whether the target person falls.

[0031] Figure 1 A flow chart of a fall monitoring method provided by an embodiment of the present application is shown. The method is executed by a computing device, which may be a computing device including one or more processors, which may be a central processing unit (CPU), or an application specific integrated circuit (ASIC), or one or more integrated circuits configured to implement an embodiment of the present invention, which is not limited here. The one or more processors included in the computing device may be processors of the same type, such as one or more CPUs; or they may be processors of different types, such as one or more CPUs and one or more ASICs, which is not limited here.

[0032] like Figure 1 As shown, the method includes the following steps:

[0033] Step 110: Continuously collect movement data of the target object.

[0034] Wherein, the target object refers to a person, an animal, or an object (such as a robot). Movement data refers to data that can be used to monitor whether the target object falls during movement. For example, the movement data can be one or more of the target object's X-axis acceleration, Y-axis acceleration, Z-axis acceleration, X-axis angular acceleration, Y-axis angular acceleration, and Z-axis angular acceleration. The following Table 1 shows common human fall classifications. Therefore, in the embodiment of the present application, preferably, the movement data includes X-axis acceleration, Y-axis acceleration, Z-axis acceleration, X-axis angular acceleration, Y-axis angular acceleration, and Z-axis angular acceleration.

[0035] Table 1

[0036]

[0037]

[0038] Specifically, the target object's movement data can be collected at a frequency that can be set as needed. The specific frequency is not limited here, as long as the target object's motion state can be monitored. The target object's movement data can be collected using a sensor, such as a six-axis inertial sensor. The sensor can be installed on a wearable device, such as a watch, or on a portable mobile terminal, such as a mobile phone.

[0039] If the target object falls in a certain direction, the acceleration and / or angular acceleration of a certain axis will change significantly compared to the normal state. Therefore, the type of motion data can be selected as needed, so that by collecting and analyzing the motion data, it is possible to monitor whether the target object has fallen.

[0040] Step 120: acquiring the movement data in at least two time periods in sequence according to the collection time sequence of the movement data, wherein the duration of each time period is the same.

[0041] The duration of each time period can be set as needed. When the total duration of mobile data collection and the frequency of mobile data collection are the same, the longer the time period is set, the more mobile data in each time period is obtained, and the fewer time periods can be obtained within the total duration of mobile data collection. For example, if the total duration of mobile data collection in step 110 is 10 seconds and the sampling interval is 0.5 seconds, a total of 20 sets of mobile data are collected. If the duration of each time period is set to 2 seconds, a maximum of 5 time periods of mobile data can be obtained, each of which includes 4 sets of mobile data. If the duration of each time period is set to 5 seconds, a maximum of 2 time periods of mobile data can be obtained, each of which includes 10 sets of mobile data.

[0042] Since step 110 continuously collects movement data of the target object to determine whether the target object has fallen at the current moment, in this step, it is preferred to obtain movement data within at least two time periods closest to the current moment. The specific number of movement data periods to be obtained can be set as needed. For example, if this step is set to obtain movement data within three time periods, the duration of each time period is set to 2 seconds. If the total duration of movement data collection in step 110 is 10 seconds, the movement data collected within the 4th to 6th seconds is obtained as the movement data within the first time period, the movement data collected within the 6th to 8th seconds is obtained as the movement data within the second time period, and the movement data collected within the 8th to 10th seconds is obtained as the movement data within the third time period.

[0043] By acquiring motion data from the time period closest to the current moment to analyze and determine whether the target subject has fallen, the accuracy of the determination can be improved compared to only acquiring motion data from time periods farther from the current moment. Furthermore, by acquiring motion data from at least two time periods to determine whether the target subject has fallen, i.e., using motion data collected at different times for analysis and determination, the accuracy of the determination can be further improved by analyzing the degree of change in the motion data to determine whether the target subject has fallen.

[0044] Step 130: Determine the feature vector of each time period based on the movement data in each time period.

[0045] If each time period acquired in step 120 includes only one piece of motion data, the feature vector for each time period in this step is composed of the motion data in that time period. If each time period acquired in step 120 includes at least two pieces of motion data, the feature vector for each time period in this step is a one-dimensional feature vector composed of the average value, variance, maximum value, minimum value, difference between the maximum and minimum values ​​of the motion data in that time period, and / or the last piece of motion data acquired in that time period.

[0046] By using the movement data in each time period to construct a feature vector, the feature vector can be input into the long short-term memory neural network model, and then the output results of the long short-term memory neural network model can be used to analyze and determine whether the target object has fallen.

[0047] Step 140: Input each eigenvector into the long short-term memory neural network model respectively, and obtain the first output result corresponding to each eigenvector respectively, wherein the first output result of the current period is determined according to the first output result of the previous period and the eigenvector of the current period. If the current period is the first period, the first output result of the previous period adopts the initialization value.

[0048] The Long Short-Term Memory (LSTM) neural network model consists of three parts: the input gate, the output gate, and the forget gate. The specific calculation formula of LSTM is as follows:

[0049] f t =σ(W fh h t-1 +W fx x t +b f );

[0050] i t =σ(W ih h t-1 +W ix x t +b i );

[0051]

[0052]

[0053] o t =σ(W oh h t-1 +W ox x t +b o );

[0054] h t =ο t tanh(c t );

[0055] Among them, f t is the forget gate at time t, σ is the activation function (i.e. sigmoid function), W fh is the weight matrix of the forget gate, h t-1 is the hidden state or unit output at time t-1, W fx is the input feature vector weight matrix of the forget gate, x t is the feature vector input to LSTM at time t, b f is the bias vector of the forget gate, i t is the input gate at time t, W ih is the weight matrix of the input gate, W ix is the input feature vector weight matrix of the input gate, b i is the bias vector of the input gate, c~ t is the candidate state of the memory unit at time t, tanh is the activation function (i.e. tanh function), W c～h is the weight matrix of the candidate state of the memory unit, W c～xis the input feature vector weight matrix of the candidate state of the memory unit, b c～ is the bias vector of the candidate state of the memory cell, c t is the state of the memory cell at time t, c t-1 is the state of the memory unit at time t-1, t is the output gate at time t, W oh is the weight matrix of the output gate, W ox is the input feature vector weight matrix of the output gate, b o is the bias vector of the output gate, h t is the hidden state or unit output at time t.

[0056] It is worth noting that before using the LSTM model for analysis, the LSTM model should be trained, and then the trained model should be used to predict and determine whether the target object has fallen. Among them, the LSTM model can be trained and verified using part of the data constructed as a feature vector in step 130. In an embodiment of the present application, preferably, the data constructed as a feature vector in step 130 is divided into a training set, a validation set, and a test set in a ratio of 7:1:2. The training set is used to train the LSTM model, the validation set is used to adjust hyperparameters and early stopping, and the test set is used to test the accuracy and generalization ability of the LSTM model. Preferably, in the process of training the LSTM model, the maximum epoch of training is equal to 1000, and an early stopping strategy is used. When the accuracy of the validation set no longer increases for 10 consecutive times during training, the training is stopped to prevent the LSTM model from overfitting. The training can use a stochastic gradient descent algorithm with momentum. Among them, the learning rate in the training parameters can be set to 0.1. Since the initial learning rate is 0.1, when the accuracy of the validation set changes by less than 0.1 for three consecutive times, the learning rate is reduced to half of the original rate. If the sampling interval of the mobile data set in step 110 is 0.25s, and since the target person usually falls for 1.5-2s, the batch size can be set to 8. The momentum can be set to 0.9, so that the network parameters can be better optimized.

[0057] The feature vectors of each time period constructed in step 130 are input into the trained LSTM model. If the feature vector currently input into the LSTM model is x t , and the corresponding output is h t , the output result corresponding to the feature vector input into the LSTM model at the last moment is h t-1 , then h t According to x t and h t-1 If the feature vector x is input to the LSTM model t is the eigenvector of the first period, then h t-1An initialization value is used, and the initialization value can be set as needed. The first output result refers to the LSTM hidden state or unit output of each time period.

[0058] Since in this step, the first output result of the current period is determined based on the first output result of the previous period and the feature vector of the current period, that is, the first output result of the current period is determined using the attention mechanism, the accuracy of the judgment is improved.

[0059] Step 150: Input the first output result corresponding to each feature vector into the attention model to obtain the second output result.

[0060] The attention model uses an additive attention mechanism. After each first output result is input into the attention model, a second output result corresponding to each first output result is obtained.

[0061] Step 160: Determine whether the target object falls according to the first output result and the second output result.

[0062] The first output result and the second output result corresponding to the feature vectors of each time period may be multiplied to obtain the product, and then the product may be normalized. Whether the target object falls may be determined based on the normalized result.

[0063] In the embodiments of the present application, by continuously collecting the target object's motion data and constructing at least two feature vectors using the motion data from at least two time periods for analysis and judgment, the number of samples analyzed is increased compared to analyzing and judging only the current moment's motion data, thereby improving the accuracy of the analysis. Furthermore, because not all collected motion data is used for analysis and judgment, but only a portion of the collected motion data is used to construct feature vectors for analysis and judgment, the amount of data analyzed is reduced compared to analyzing all collected motion data, thereby improving the speed of analysis and judgment.

[0064] Then, each feature vector is input into the long short-term memory neural network model to obtain the first output result corresponding to each vector. Since the long short-term memory neural network model uses the attention mechanism to obtain the first output result of the current period, that is, the first output result of the current period is determined based on the first output result of the previous period and the feature vector of the current period, the accuracy of the obtained first output result is improved. Each first output result is then input into the attention model to obtain the corresponding second output result. Finally, based on the first output result and the second output result, it is determined whether the target object has fallen. Since the speed of obtaining the first output result is fast and the accuracy is high, the speed of further using the first output result to determine whether the target object has fallen is also fast, and the accuracy of the obtained judgment result is also high.

[0065] In order to reduce the computational complexity and power consumption of LSTM models and attention models, Figure 2 FIG2 shows a flow chart of a fall monitoring method provided by another embodiment of the present application. The embodiment of the present application is based on the above embodiment, and adds a step of determining whether the target object has pre-fall between step 110 and step 120. Figure 2 As shown, the method includes the following steps:

[0066] Step 210: Continuously collect movement data of the target object.

[0067] Among them, step 210 is the same as step 110. Therefore, the specific implementation method of step 210 can refer to step 110 and will not be repeated here.

[0068] Step 220: Obtain movement data within a preset time period.

[0069] Whether the target object has pre-fall is determined by obtaining movement data within a preset time period, wherein the preset time period can be set as needed and is not limited here. In the embodiment of the present application, the preset time period is set to 0.25s. Specifically, if it is necessary to determine whether the target object has pre-fall at the current moment, the movement data collected within the 0.25s closest to the current moment is obtained, thereby improving the accuracy of the judgment. For example, if the total time period for collecting movement data up to the current moment in step 210 is 10s, the movement data collected within 0.75-10s is obtained.

[0070] Step 230: Pre-fall determination is performed on the target subject based on the movement data within a preset time period to determine whether the target subject has pre-falled. If the target subject has pre-falled, step 240 is executed; if the target subject has not pre-falled, the process proceeds to step 220.

[0071] Before a fall, the target object will tilt in a certain direction, which is called a pre-fall. If the target object pre-falls, its movement data will change significantly. Therefore, the change amount or change rate of the movement data within a preset time period can be used to determine whether the target object has pre-falled.

[0072] Step 240: Continue to collect at least one movement data.

[0073] If the target object is determined to be in a pre-fall state in step 230, at least one more movement data set is continuously collected to further determine whether the target object has fallen. Since a fall typically lasts 1.5-2 seconds, continuing to collect the target object's movement data increases the number of data samples analyzed, thereby improving the accuracy of the determination.

[0074] Step 250: acquiring movement data in at least two time periods in sequence according to the collection time sequence of the movement data, wherein the acquired movement data includes at least one movement data acquired after determining that the target object has pre-fallen.

[0075] The duration of each time period can be set as needed. When the total duration of mobile data collection and the frequency of mobile data collection are the same, the longer the time period is set, the more mobile data in each time period is obtained, and the fewer time periods can be obtained within the total duration of mobile data collection. For example, if the total duration of mobile data collection in step 210 is 10 seconds and the sampling interval is 0.5 seconds, a total of 20 sets of mobile data are collected. If the duration of each time period is set to 2 seconds, a maximum of 5 time periods of mobile data can be obtained, each of which includes 4 sets of mobile data. If the duration of each time period is set to 5 seconds, a maximum of 2 time periods of mobile data can be obtained, each of which includes 10 sets of mobile data.

[0076] Since step 240 continues to collect the target object's movement data after determining that the target object has pre-falled, this step further determines whether the target object has fallen by acquiring movement data from at least two time periods. The acquired movement data includes at least one movement data acquired in step 240, thereby improving the accuracy of the determination. In this step, it is preferred to acquire movement data from at least two time periods closest to the current moment. The specific number of movement data acquired can be set as needed. For example, if this step is set to acquire movement data from three time periods, each time period is set to 2 seconds. If the total time period for acquiring movement data in step 210 is 10 seconds, after step 230 determines that the target object has pre-falled, step 240 continues to acquire movement data for 1 second (i.e., acquiring movement data up to the 11th second), then in this step, the movement data acquired from the 5th to 7th second is acquired as the movement data for the first time period, the movement data acquired from the 7th to 9th second is acquired as the movement data for the second time period, and the movement data acquired from the 9th to 11th second is acquired as the movement data for the third time period.

[0077] Step 260: Determine the feature vector of each time period based on the movement data in each time period.

[0078] Among them, step 260 is the same as step 130. Therefore, the specific implementation method of step 260 can refer to step 130 and will not be repeated here.

[0079] Step 270: Input each eigenvector into the long short-term memory neural network model respectively, and obtain the first output result corresponding to each eigenvector respectively, wherein the first output result of the current period is determined according to the first output result of the previous period and the eigenvector of the current period. If the current period is the first period, the first output result of the previous period adopts the initialization value.

[0080] Step 280: Input the first output result corresponding to each feature vector into the attention model to obtain the second output result.

[0081] Step 290: Determine whether the target object falls according to the first output result and the second output result.

[0082] Among them, steps 270 to 290 are the same as steps 140 to 160. Therefore, the specific implementation of steps 270 to 290 can refer to steps 140 to 160 and will not be repeated here.

[0083] This application constructs a feature vector and inputs it into the LSTM model to obtain a first output result, then inputs the first output result into the attention model to obtain a second output result, and finally determines whether the target object has fallen based on the first output result and the second output result. However, the LSTM model and the attention model have a large amount of computation. Therefore, if the LSTM model and the attention model are used to determine whether the target object has fallen after each movement data is collected in step 210, it is easy to cause the LSTM model and the attention model to have a large amount of computation and power consumption.

[0084] In an embodiment of the present application, by utilizing mobile data to determine whether the target object has pre-fallen, if it is determined that a pre-fall has occurred, a feature vector is constructed and the LSTM model and the attention model are used to further determine whether the target object has fallen, thereby reducing the number of times the feature vector is constructed and the number of times the LSTM model and the attention model are used for calculation, thereby reducing the amount of calculation of the LSTM model and the attention model, and reducing the power consumption requirements of the execution device.

[0085] In order to improve the accuracy of determining whether the target object has pre-fall, step 230 further includes:

[0086] Step a1: Obtain the maximum change in the movement data within a preset time period.

[0087] Among them, if the movement data collected in step 210 is one of the acceleration in the X-axis direction, the acceleration in the Y-axis direction, the acceleration in the Z-axis direction, the angular acceleration in the X-axis direction, the angular acceleration in the Y-axis direction, and the angular acceleration in the Z-axis direction, then a first-order difference operation is performed on the movement data within the preset time length, and the maximum value in the first-order difference operation result is the maximum change in the movement data.

[0088] If the movement data collected in step 210 includes at least two types of data among the X-axis acceleration, the Y-axis acceleration, the Z-axis acceleration, the X-axis angular acceleration, the Y-axis angular acceleration, and the Z-axis angular acceleration, a first-order difference operation is first performed on each type of movement data, and then the maximum value of the first-order difference operation results in each type of movement data is obtained. Finally, the maximum value is selected from the maximum values ​​of the first-order difference operation results in all the movement data. The maximum value finally selected is the maximum change in the movement data.

[0089] Wherein a2: If the maximum change is greater than a preset threshold, it is determined that the target object is about to fall.

[0090] The preset threshold value can be set as needed and is not limited here.

[0091] In an embodiment of the present application, since the target object will tilt in a certain direction before falling during movement, that is, the movement data in this direction will change significantly compared to normal circumstances, therefore, by determining whether the maximum change in the movement data is greater than a threshold, it is possible to accurately determine whether the target object is about to fall.

[0092] In order to improve the accuracy of judging whether the target object has fallen, Figure 3 FIG. 1 shows a flow chart of a fall monitoring method provided by another embodiment of the present application. The embodiment of the present application is based on the above embodiment, and adds a step of pre-processing the movement data between step 120 and step 130, such as Figure 3 As shown, the method includes the following steps:

[0093] Step 310: Continuously collect movement data of the target object.

[0094] Step 320: acquiring the movement data in at least two time periods in sequence according to the collection time sequence of the movement data, wherein the duration of each time period is the same.

[0095] Among them, steps 310 to 320 are the same as steps 110 to 120. Therefore, the specific implementation of steps 310 to 320 can refer to steps 110 to 120 and will not be repeated here.

[0096] Step 330: Preprocess the movement data in each time period to remove baseline drift and / or noise, and obtain preprocessed movement data.

[0097] In this step, morphological methods can be used to remove baseline drift. Specifically, a morphological opening operation is performed on the input signal, followed by a morphological closing operation to obtain a baseline signal. The baseline signal is then subtracted from the original input signal to obtain a signal free of baseline drift. The morphological opening operation involves first performing an erosion operation on the signal, followed by a dilation operation; the morphological closing operation involves first performing a dilation operation on the signal, followed by an erosion operation. The input signal refers to the motion data collected in step 310.

[0098] In this step, a low-pass filter can be used to remove high-frequency noise. To ensure the operating efficiency of the device, an IIR low-pass filter can be designed by converting an elliptical analog filter to a digital filter. Specifically, the parameters of the analog filter can be determined based on the PPG signal, and then an elliptical analog filter can be designed based on these parameters. Finally, the analog filter can be converted to an IIR digital filter. Because the elliptical low-pass filter has equal ripples in the passband and stopband, the passband and stopband approximation characteristics are good. In the embodiment of the present application, an elliptical low-pass filter is used to design an IIR low-pass filter, thereby improving the effect of removing high-frequency noise.

[0099] Step 340: Determine a feature vector based on the pre-processed motion data in each time period.

[0100] If each time period in step 330 includes only one piece of preprocessed motion data, then the feature vector for each time period in this step is composed of the preprocessed motion data in that time period. If each time period in step 330 includes at least two pieces of preprocessed motion data, then the feature vector for each time period in this step can be a one-dimensional feature vector composed of the average value, variance, maximum value, minimum value, difference between the maximum value and minimum value of the preprocessed motion data in that time period, and / or the last motion data collected in that time period.

[0101] By using the preprocessed movement data in each time period to construct a feature vector, the feature vector can be input into the long short-term memory neural network model, and then the output results of the long short-term memory neural network model can be used to analyze and determine whether the target object has fallen.

[0102] Step 350: Input each feature vector into the long-short-term memory neural network model to obtain the first output result corresponding to each feature vector. The first output result of the current period is determined based on the first output result of the previous period and the feature vector of the current period. If the current period is the first period, the first output result of the previous period uses the initialization value. Step 360: Input the first output result corresponding to each feature vector into the attention model to obtain the second output result. Step 370: Determine whether the target object has fallen based on the first and second output results.

[0103] Among them, steps 350 to 370 are the same as steps 140 to 160. Therefore, the specific implementation of steps 350 to 370 can refer to steps 140 to 160 and will not be repeated here.

[0104] Because the motion data collected by the sensor contains a lot of noise and baseline drift, if it is not processed to remove baseline drift and / or denoise, the accuracy of using the motion data to determine whether the target object has fallen will decrease. Therefore, in the embodiments of the present application, by using the motion data after removing baseline drift and / or denoising to determine whether the target object has fallen, the accuracy of the determination can be improved.

[0105] In order to further improve the accuracy of determining whether the target object has fallen, the embodiment of the present application provides a method for constructing a feature vector. In the embodiment of the present application, Figure 4 Shown Figure 1 Flowchart of sub-steps of step 130. Figure 4 As shown, step 130 determines the feature vector of each time period based on the movement data in each time period, further comprising:

[0106] Step 131: Determine the average value, variance, maximum value, minimum value, and / or the difference between the maximum value and the minimum value in the movement data within each time period.

[0107] If each time period contains multiple different types of motion data, the average, variance, maximum, minimum, and / or difference between the maximum and minimum values ​​for each type of motion data are determined. For example, if each time period contains the target object's X-axis acceleration, Y-axis acceleration, Z-axis acceleration, X-axis angular acceleration, Y-axis angular acceleration, and Z-axis angular acceleration, the average, variance, maximum, minimum, and / or difference between the maximum and minimum values ​​for the acceleration and angular acceleration along each axis are determined.

[0108] Step 132: Perform a fast Fourier transform on the mobile data in each time period to obtain a fast Fourier transform result.

[0109] A fast Fourier transform is performed on each of the time-domain mobile data within each time period obtained in step 120 to obtain a fast Fourier transform result corresponding to the mobile data within each time period. By performing a fast Fourier transform on the time-domain mobile data, i.e., converting the time-domain mobile data into frequency-domain data, data features not reflected in the time-domain mobile data can be obtained by analyzing the frequency-domain data, thereby more comprehensively obtaining the features of the mobile data and improving the accuracy of the judgment.

[0110] Step 133: Obtain the frequency value corresponding to the maximum amplitude value in the fast Fourier transform result in each time period.

[0111] According to the fast Fourier transform results in each time period obtained in step 132 , the frequency value corresponding to the maximum amplitude in the fast Fourier transform results corresponding to all the time-domain motion data in each time period is determined.

[0112] Step 134: construct a feature vector based on the average value, variance, difference, frequency value corresponding to the maximum amplitude in each time period and / or the last collected movement data in the time period.

[0113] The feature vector may be constructed based on the average value, variance, difference, and frequency value corresponding to the maximum amplitude value within each time period determined in steps 131 and 133, and / or the last movement data collected within the time period. In the embodiment of the present application, preferably, the feature vector is constructed based on the average value, variance, difference, and frequency value corresponding to the maximum amplitude value within each time period, and the last movement data collected within the time period, thereby improving the accuracy of the judgment.

[0114] In the embodiment of the present application, by constructing a feature vector using the average value, variance, difference, and frequency value corresponding to the maximum amplitude within each time period, along with the last collected mobile data within the time period, both time-domain and frequency-domain data are included, thereby comprehensively capturing the characteristics of the mobile data and improving the accuracy of the judgment. Furthermore, because the average value, variance, and difference between the maximum and minimum values ​​within each time period accurately reflect the changes in the mobile data within that time period, constructing a feature vector using this data reduces the amount of data in the feature vector while still reflecting the degree of change in all mobile data within the time period, compared to constructing a feature vector using all mobile data within the time period. This improves the speed of analysis and judgment without compromising the accuracy of the judgment.

[0115] In order to further improve the accuracy of determining whether the target object has fallen, the embodiment of the present application provides a method for determining whether the target object has fallen based on the first output result and the second output result. In the embodiment of the present application, Figure 5 Shown Figure 1 Flowchart of sub-steps of step 160. Figure 5As shown, step 160 determines whether the target object falls according to the first output result and the second output result, further comprising:

[0116] Step 161: Multiply the first output result and the second output result corresponding to each eigenvector to obtain the product corresponding to each eigenvector.

[0117] The first output result corresponding to each eigenvector obtained in step 140 and the second output result corresponding to each first output result obtained in step 150 are cross-multiplied to obtain the product corresponding to each eigenvector.

[0118] Specifically, if there are m eigenvectors determined in step 130, there are correspondingly m first output results h i , where i = 1, 2, ..., m. The m first output results are used as the columns of the matrix to construct a matrix H of size 1×m 1×m :

[0119] H 1×m =[h1,h2,…h m ];

[0120] The second output result A i , where i = 1, 2, ..., m, is calculated according to the following formula:

[0121] E 1×m =tanh(W*H 1×m +b);

[0122]

[0123] Among them, W and b are the weight matrix and bias vector of the attention model respectively.

[0124] Then the first output result h i and the second output result A i Perform a cross product.

[0125] Step 162: Add the products corresponding to the eigenvectors to obtain a sum.

[0126] Specifically, the products corresponding to the eigenvectors obtained in step 161 are added together to obtain the Out matrix:

[0127]

[0128] Step 163: Input the sum into the multilayer perceptron to obtain a third output result.

[0129] Among them, the multilayer perceptron (MLP) consists of two fully connected layers and a Relu activation function, and can be constructed using Dense and Dropout in the tensorflow framework.

[0130] Step 164: Input the third output result into the normalized exponential function layer, and determine whether the target object falls according to the normalized processing result of the normalized exponential function layer.

[0131] Among them, the normalized exponential function layer, namely the Softmax layer, consists of the softmax function and the cross-entropy loss function. It can convert the vector data input by the fully connected layer of the multi-layer perceptron into a set of probability value vectors that reflect the classification results, so that it can be judged whether the target object has fallen based on the output results of the Softmax layer.

[0132] Since the Softmax layer can directly judge whether the target object has fallen based on the input data, thereby improving the efficiency of judgment, in an embodiment of the present application, the first output result and the second output result are cross-multiplied to obtain the product, and then the products are added together and input into the MLP layer, and finally the output result of the MLP layer is input into the Softmax layer, so that the judgment result of whether the target object has fallen can be directly obtained, thereby improving the speed of judgment.

[0133] The computing device that executes the fall monitoring method provided in the embodiments of the present application can be a smart wearable device (such as a watch, bracelet, smart glasses, head cover, ring, etc.), that is, steps 110-160 are all executed by the smart wearable device; or it can be a mobile terminal (such as a mobile phone), that is, steps 110-160 are all executed by the mobile terminal. In some embodiments, in order to improve the efficiency of determining whether the target object has fallen, steps 110-130 are all executed by the smart wearable device, and steps 140-160 are all executed by the mobile terminal associated with the smart wearable device. The following steps are also included between steps 130 and 140:

[0134] The smart wearable device transmits the feature vector to the mobile terminal.

[0135] Smart wearable devices, such as smartwatches, are wearable devices that can collect movement data from target objects. Mobile terminals can be mobile phones, computers, and other devices. The smart wearable device can transmit feature vectors to the mobile terminal using Bluetooth, mobile communications, Wi-Fi, and other transmission methods.

[0136] Specifically, the deep learning model file obtained by training the LSTM model with the training set in step 140 can be converted into a Tensorflow Lite model file used by the mobile terminal through the Tensorflow Lite library function interface, and then the Tensorflow Lite file can be parsed using the inference library of the mobile terminal to perform deep learning model inference.

[0137] Since the LSTM model requires a large amount of computation, smart wearable devices that have strict requirements on power consumption cannot use the LSTM model. In the embodiment of the present application, mobile data is collected and a feature vector is constructed by the smart wearable device, and then the constructed feature vector is transmitted to the mobile terminal. The mobile terminal then uses the LSTM model and the attention model to analyze and determine whether the target object has fallen, which greatly reduces the power consumption requirements of the smart wearable device, thereby making the embodiment of the present application more widely applicable to various types of smart wearable devices. At the same time, since the storage capacity and computing speed of the mobile terminal are superior to those of the smart wearable device, the analysis speed can be improved by using the model through the mobile terminal compared to the analysis through the model through the smart wearable device. Furthermore, if a large amount of mobile data is collected in step 110, and step 130 only uses part of the mobile data collected in step 110 to construct a feature vector, then the amount of data of the feature vector constructed in step 130 is less than the amount of data collected in step 110. Therefore, constructing the feature vector through the smart wearable device and then transmitting it to the mobile terminal, compared to transmitting all the collected data to the mobile terminal, can increase the speed of data transmission due to the reduction in the amount of transmitted data, thereby further improving the efficiency of determining whether the target object has fallen.

[0138] In some embodiments, if it is determined in step 160 that the target object has fallen, an alarm message is immediately generated and sent to the mailbox or smart device of a pre-associated contact via email or text message, so that the contact can receive the alarm message and rescue the fallen person in time to avoid missing the best rescue time.

[0139] Figure 6 A schematic structural diagram of a fall monitoring device provided in an embodiment of the present application is shown. The specific embodiments of the present application do not limit the specific implementation of the fall monitoring device.

[0140] like Figure 6 As shown, the fall monitoring device may include: a processor 402 , a communications interface 404 , a memory 406 , and a communication bus 408 .

[0141] Processor 402, communication interface 404, and memory 406 communicate with each other via communication bus 408. Communication interface 404 is used to communicate with other devices, such as clients or other server network elements. Processor 402 is used to execute program 410, which may specifically perform the steps described in the above-mentioned embodiment of the fall monitoring method.

[0142] Specifically, the program 410 may include program code including computer-executable instructions.

[0143] The processor 402 may be a central processing unit (CPU), an application-specific integrated circuit (ASIC), or one or more integrated circuits configured to implement the embodiments of the present application. The one or more processors included in the fall monitoring device may be processors of the same type, such as one or more CPUs, or may be processors of different types, such as one or more CPUs and one or more ASICs.

[0144] The memory 406 is used to store the program 410. The memory 406 may include a high-speed RAM memory, and may also include a non-volatile memory (non-volatile memory), such as at least one disk memory.

[0145] Figure 7 FIG. 1 shows a schematic diagram of the structure of the fall monitoring system provided in an embodiment of the present application. Figure 7 As shown, the system 500 includes a smart wearable device 501 and a mobile terminal 502.

[0146] The smart wearable device 501 is configured to continuously collect movement data of a target object; and to sequentially obtain movement data of at least two time periods according to a time sequence of movement data collection, wherein each time period has the same duration; and further configured to determine a feature vector of each time period based on the movement data of each time period;

[0147] The mobile terminal 502 is used to input each feature vector into the long short-term memory neural network model respectively, and obtain the first output result corresponding to each feature vector respectively, wherein the first output result of the current time period is determined based on the first output result of the previous time period and the feature vector of the current time period. If the current time period is the first time period, the first output result of the previous time period adopts the initialization value; and is used to input the first output result corresponding to each feature vector into the attention model to obtain the second output result; and is also used to determine whether the target object falls based on the first output result and the second output result.

[0148] Since the LSTM model has a large amount of calculation, smart wearable devices that have strict requirements on power consumption cannot use the LSTM model. In the fall monitoring system 500 provided in this application, the mobile data is collected and a feature vector is constructed through the smart wearable device 501, and the constructed feature vector is then transmitted to the mobile terminal 502. The mobile terminal 502 then uses the LSTM model and the attention model to analyze and determine whether the target object has fallen, which greatly reduces the power consumption requirements of the smart wearable device 501, thereby making this application more widely applicable to various types of smart wearable devices 501. At the same time, since the storage capacity and computing speed of the mobile terminal 502 are superior to those of the smart wearable device 501, the speed of analysis and calculation can be improved by using the model for analysis and calculation through the mobile terminal 502, compared with the analysis and calculation by using the model through the smart wearable device 501. Furthermore, if the smart wearable device 501 collects a large amount of mobile data, but only uses part of the collected mobile data to construct a feature vector, the amount of data in the constructed feature vector is less than the amount of collected data. Therefore, after constructing the feature vector through the smart wearable device 501 and then transmitting it to the mobile terminal 502, compared with transmitting all the collected data to the mobile terminal 502, the amount of transmitted data is reduced, thereby increasing the speed of data transmission, thereby further improving the efficiency of determining whether the target object has fallen.

[0149] In an optional manner, the smart wearable device 501 is also used to execute all the steps in the above-mentioned fall monitoring method embodiment.

[0150] In an optional manner, if the mobile terminal 502 is placed on the target object and can collect movement data of the target object, the mobile terminal 502 is also used to execute all the steps in the above-mentioned fall monitoring method embodiment.

[0151] In an optional manner, the mobile terminal 502 is also used to execute the relevant steps in the above-mentioned fall monitoring method embodiment. If the mobile terminal 502 cannot collect the movement data of the target object, the movement data processed by the mobile terminal 502 comes from the movement data collected by the smart wearable device 501.

[0152] An embodiment of the present application provides a computer-readable storage medium storing executable instructions. When the executable instructions are executed on a fall monitoring device, the fall monitoring device executes the fall monitoring method in any of the above method embodiments.

Claims

1. A fall monitoring method, characterized in that: The method comprises: Continuously collect movement data of target objects; Sequentially acquiring the movement data within at least two time periods according to a time sequence of the movement data collection, wherein each of the time periods has the same length; determining a feature vector for each time period according to the movement data within each time period; Inputting each of the feature vectors into a long short-term memory neural network model, respectively, to obtain a first output result corresponding to each of the feature vectors, respectively, wherein the first output result of the current period is determined based on the first output result of the previous period and the feature vector of the current period. If the current period is the first period, the first output result of the previous period uses an initialization value; Inputting the first output result corresponding to each of the feature vectors into an attention model to obtain a second output result; determining whether the target object falls according to the first output result and the second output result; Before sequentially acquiring the movement data in at least two time periods according to the collection time sequence of the movement data, the method further includes: Acquiring the movement data within a preset time period; Performing a fall pre-judgment on the target object based on the movement data within the preset time period to determine whether the target object has pre-fallen; If it is determined that the target object is about to fall, continue to collect at least one movement data; The acquiring the movement data in at least two time periods in sequence according to the collection time sequence of the movement data further includes: The movement data within at least two time periods are acquired in sequence according to the acquisition time sequence of the movement data, wherein the acquired movement data includes at least one movement data acquired after it is determined that the target object has pre-fallen.

2. The method according to claim 1, characterized in that The pre-judgment of the target object falling according to the movement data within the preset time period, and determining whether the target object has pre-fallen, further includes: Obtaining a maximum change in the movement data within the preset time period; If the maximum change is greater than a preset threshold, it is determined that the target object is about to fall.

3. The method according to claim 1, characterized in that Before determining the feature vector of each time period according to the movement data in each time period, the method further includes: performing preprocessing on the movement data in each of the time periods to remove baseline drift and / or noise, to obtain preprocessed movement data; The determining of the feature vector of each time period according to the movement data in each time period further includes: A feature vector is determined according to the pre-processed movement data in each of the time periods.

4. The method according to claim 1, wherein The determining of the feature vector of each time period according to the movement data in each time period further includes: determining an average value, a variance, a maximum value, a minimum value, and / or a difference between the maximum value and the minimum value in the movement data within each of the time periods; Performing a fast Fourier transform on the movement data in each of the time periods to obtain a fast Fourier transform result; Obtaining the frequency value corresponding to the maximum amplitude value in the fast Fourier transform result within each of the time periods; The feature vector is constructed according to the average value, the variance, the difference, the frequency value corresponding to the maximum amplitude in each time period and / or the movement data collected last in the time period.

5. The method according to claim 1, wherein The determining whether the target object falls according to the first output result and the second output result further includes: multiplying the first output result and the second output result corresponding to each of the eigenvectors to obtain a product corresponding to each of the eigenvectors; Adding the products corresponding to the eigenvectors to obtain a sum; Inputting the sum into a multilayer perceptron to obtain a third output result; The third output result is input into a normalized exponential function layer, and whether the target object falls is determined according to a normalized processing result of the normalized exponential function layer.

6. The method according to claim 1, characterized in that The steps of determining the feature vector of each time period based on the movement data in each time period and the steps preceding the step are performed by the smart wearable device, the steps of inputting each feature vector into the long short-term memory neural network model to obtain the first output result corresponding to each feature vector and the steps following the step are performed by a mobile terminal associated with the smart wearable device, and after determining the feature vector of each time period based on the movement data in each time period, the method further includes: The smart wearable device transmits the feature vector to the mobile terminal.

7. A fall monitoring device, characterized in that: include: A processor, a memory, a communication interface, and a communication bus, wherein the processor, the memory, and the communication interface communicate with each other via the communication bus; The memory is used to store at least one executable instruction, and the executable instruction enables the processor to execute the fall monitoring method according to any one of claims 1 to 6.

8. A fall monitoring system, characterized in that: The system includes a smart wearable device and a mobile terminal associated with the smart wearable device: The smart wearable device is used to continuously collect movement data of a target object; obtain the movement data within a preset time period, perform a pre-fall judgment on the target object based on the movement data within the preset time period, and judge whether the target object has pre-fall; if it is judged that the target object has pre-falled, continue to collect at least one movement data, and is used to sequentially obtain the movement data within at least two time periods according to the collection time sequence of the movement data, wherein the time period of each of the movement data is the same, and the movement data obtained includes at least one movement data collected after the target object is judged to have pre-falled; and is further used to determine a feature vector for each time period based on the movement data within the time period; The mobile terminal is used to input each of the feature vectors into a long short-term memory neural network model respectively, and obtain a first output result corresponding to each of the feature vectors respectively, wherein the first output result of the current time period is determined based on the first output result of the previous time period and the feature vector of the current time period. If the current time period is the first time period, the first output result of the previous time period adopts an initialization value; and is used to input the first output result corresponding to each of the feature vectors into an attention model to obtain a second output result; and is also used to determine whether the target object has fallen based on the first output result and the second output result.

9. A computer-readable storage medium, characterized in that The storage medium stores at least one executable instruction, and the executable instruction enables the processor to execute the fall monitoring method according to any one of claims 1 to 6.

Images