A human fall detection method and system based on time series characteristics

Abstract

The present invention discloses a method and system for detecting human falls based on time series features, belonging to the technical field of abnormal behavior detection using computer vision. The method comprises the following steps: obtaining video samples; decomposing and comparing the video samples frame by frame, extracting time series features of arm angles and time series features of entropy of head skeletal points; inputting the two sets of time series features into a pre-established neural network to obtain a first output tensor and a second output tensor containing abnormal frame information, fusing the obtained feature tensor, and inputting the obtained feature tensor into a classifier composed of a fully connected layer and a Sigmoid function for classification. The classifier has been trained in advance and outputs the probability of a human fall or non-fall, with the one with the higher probability being the detection result. By fusing the time series features of the arm angles and the time series features of the entropy of the head skeletal points, human behavior can be better represented, thereby improving the probability of detecting a human fall.

Description

Technical Field

[0001] The present invention relates to a method and system for detecting human falls based on time series features, and belongs to the technical field of abnormal behavior detection using computer vision. Background Art

[0002] In daily life, there are two main reasons why people fall. One is tripping or slipping due to inconvenient legs and feet, and the other is falling due to illness. If you do not receive timely help when you fall, it will often lead to worsening injuries or even cost your life. Therefore, it is particularly important to detect human falls.

[0003] At present, common human fall detection technologies can be divided into three categories, including wearable-based, environmental sensor-based and video-based. The wearable-based method is to place the sensor in a belt or watch, etc., but the elderly may forget to wear it due to memory loss or not wear it because of discomfort. The environmental sensor-based method, such as infrared technology monitoring, but some people are allergic to infrared rays, which affects human health. The video-based method currently combines the human skeleton to detect fall behavior, including the displacement of head bone points and changes in leg bone points, but the legs are often not the first to react when a person falls, so there is a delay in detection. Summary of the Invention

[0004] The purpose of the present invention is to provide a human fall detection method and system based on time series features. By extracting the time series features of the arm angles, the human body's behavior can be better represented, thereby making human fall detection more accurate. By extracting the time series features of the entropy of the head bone points, the degree of chaos of the human body movement can be obtained, which better helps to detect the probability of a human fall.

[0005] To achieve the above objectives, the present invention is implemented by adopting the following technical solutions:

[0006] In a first aspect, the present invention provides a method for detecting a human fall based on time series features, comprising:

[0007] Get video samples;

[0008] Decompose and compare the video samples frame by frame to extract the temporal features of the arm angle and the temporal features of the entropy of the head bone points;

[0009] Input the time series features of the arm angle and the time series features of the entropy of the head bone points into a pre-established neural network to obtain a first output tensor and a second output tensor containing abnormal frame information. After fusing the first output tensor and the second output tensor, a feature tensor is obtained.

[0010] The feature tensor is input into a classifier consisting of a fully connected layer and a Sigmoid function for classification. The classifier has been trained in advance and outputs the probability of a person falling or not falling in the abnormal frame. The one with the highest probability is the detection result.

[0011] In combination with the first aspect, the video samples are further decomposed and compared frame by frame, including:

[0012] The human body motion area is obtained through the changes in the optical flow field to achieve dynamic human body tracking. The coordinates of the arm bone points in the video sample are obtained through the OpenPose algorithm. The angle features of the arm are calculated by the coordinates of the arm bone points. The angle features of the arm are the angle features of the upper and lower arms. The angle features of the arm in each frame of the video sample are compared with the previous frame to obtain the angle timing features of the arm.

[0013] In combination with the first aspect, further decomposing and comparing the video samples frame by frame also includes:

[0014] The human body motion area is obtained through changes in the optical flow field to achieve dynamic human body tracking. The coordinates of the head bone points in the video sample are obtained through the OpenPose algorithm. The movement direction and amplitude of the head bone points are calculated, and the entropy of the movement direction and the entropy of the amplitude are calculated. The entropy of the movement direction and the entropy of the amplitude are multiplied together to obtain the entropy of the head bone points. The entropy of the head bone points in each frame of the video sample is compared with the previous frame to obtain the temporal characteristics of the entropy of the head bone points.

[0015] In combination with the first aspect, the method further includes the step of preprocessing the video sample:

[0016] The RGB channel images in the video samples are converted into HSL channel images to solve the impact of unstable lighting on the optical flow field. The conversion is performed using the following formula:

[0017]

[0018]

[0019]

[0020] Where H, S, and L correspond to hue, saturation, and lightness, respectively; R, G, and B correspond to the brightness of red, green, and blue, respectively; arccos represents the inverse cosine function; and min returns the minimum value in the given parameter table.

[0021] In combination with the first aspect, further, the feature tensor is obtained by the following method:

[0022] The angle timing features of the arm and the entropy timing features of the head bone points are input into a pre-established neural network, which includes an LSTM-based angle timing feature attention module and an entropy timing feature attention module. The angle timing features of the arm and the entropy timing features of the head bone points are respectively input into the angle timing feature attention module and the entropy timing feature attention module to obtain a first output tensor and a second output tensor containing abnormal frame information. The first output tensor and the second output tensor are fused into a feature tensor using the Concat function.

[0023] In combination with the first aspect, further, the classifier is pre-trained using a binary cross entropy loss function, and the binary cross entropy loss function is:

[0024]

[0025] Among them, Z is the binary cross entropy loss, N is the number of samples, a represents the counter, a=1 means counting from the first sample, is the label of the sample, the positive class is 1, the negative class is 0, y a is the probability of the prediction being positive.

[0026] Combined with the first aspect, the calcOpticalFlowFarneBack operator in OpenCV is further used to calculate the change of the optical flow field between two frames, and the motion area is obtained according to the change of the optical flow field to realize human body dynamic tracking.

[0027] In a second aspect, the present invention further provides a human fall detection system based on time series features, comprising:

[0028] Input module: used to obtain video samples;

[0029] Temporal feature extraction module: used to decompose and compare video samples frame by frame, extracting the temporal features of arm angles and the temporal features of the entropy of head bone points;

[0030] Feature tensor output module: used to input the time series features of the arm angle and the time series features of the entropy of the head bone points into the pre-established neural network, obtain the first output tensor and the second output tensor containing the abnormal frame information, and obtain the feature tensor after fusing the first output tensor and the second output tensor;

[0031] Detection result classification module: It is used to input the feature tensor into a classifier composed of a fully connected layer and a Sigmoid function for classification. The classifier has been trained in advance and outputs the probability of a person falling or not falling in the abnormal frame. The one with the higher probability is the detection result.

[0032] Compared with the prior art, the present invention has the following beneficial effects:

[0033] The present invention provides a method and system for detecting human falls based on time series features, which decompose and compare video samples frame by frame to extract the angle time series features of the arms and the time series features of the entropy of the head bone points. By extracting the angle time series features of the arms, that is, the angle features of the arms compared frame by frame, when a human falls, the changes in the arms are the most obvious and rapid. By extracting the angle time series features of the arms, the behavior of the human body can be better represented, thereby making human fall detection more accurate. By extracting the time series features of the entropy of the head bone points, that is, the changes in the entropy of the head bone points compared frame by frame, the degree of confusion of the human body movement can be obtained. The larger the entropy value, the more chaotic the movement and the greater the probability of falling, which can better help detect the probability of human falls. The data are then input into a pre-established neural network to obtain a first output tensor and a second output tensor containing abnormal frame information of the fall process. After fusion, a feature tensor is obtained, and the feature tensor is input into a classifier for classification to obtain a detection result to determine whether the human body has fallen.

[0034] The pre-established neural network includes an LSTM (Long Short-Term Memory)-based angle temporal feature attention module and an entropy temporal feature attention module. The angle temporal features of the arm and the entropy temporal features of the head bone points are respectively input into the angle temporal feature attention module and the entropy temporal feature attention module to obtain the first output tensor and the second output tensor containing abnormal frame information. The use of two LSTM-based temporal attention modules can better detect abnormal frames in video samples and then determine whether the human body has fallen. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] Figure 1 This is one of the flow charts of a human fall detection method based on time series features provided by an embodiment of the present invention;

[0036] Figure 2 This is a second flow chart of a human fall detection method based on time series features provided by an embodiment of the present invention;

[0037] Figure 3 Schematic diagram of the processing of the neural network and classifier provided by the embodiment of the present invention. DETAILED DESCRIPTION

[0038] The present invention will be further described below in conjunction with the accompanying drawings. The following embodiments are only used to more clearly illustrate the technical solutions of the present invention and are not intended to limit the scope of protection of the present invention.

[0039] Example 1

[0040] like Figure 1 As shown, an embodiment of the present invention provides a human fall detection method based on time series features, including:

[0041] S1. Obtain video samples.

[0042] In home monitoring, a surveillance video is selected as a video sample. The video sample is first preprocessed to convert the RGB channel image in the video sample into the HSL channel image to solve the impact of unstable lighting on the optical flow field. The conversion is performed using the following formula:

[0043]

[0044]

[0045]

[0046] Where H, S, and L correspond to hue, saturation, and lightness, respectively; R, G, and B correspond to the brightness of red, green, and blue, respectively; arccos represents the inverse cosine function; and min returns the minimum value in the given parameter table.

[0047] S2. Decompose and compare the video samples frame by frame to extract the temporal features of the arm angle and the temporal features of the entropy of the head bone points.

[0048] The video samples are decomposed and compared frame by frame. First, the coordinates of the arm bone points and the head bone points of the human body are collected through the OpenPose algorithm. Then, the angle features of the upper and lower arms are calculated through the arm bone point coordinates. The angle features of the arms in each frame of the video sample are compared with the angle features of the arms in the previous frame to obtain the angle time series features of the arms. At the same time, the motion direction and amplitude are calculated through the coordinates of the head bone points. Then, the entropy of the motion direction and the entropy of the amplitude are calculated, and the two are multiplied to obtain the entropy of the head bone points. The entropy of the head bone points in each frame of the video sample is compared with the entropy of the head bone points in the previous frame to obtain the time series features of the entropy of the head bone points.

[0049] S3. Input the temporal features of the arm angle and the temporal features of the entropy of the head bone points into a pre-established neural network, output a first output tensor and a second output tensor containing abnormal frame information, and obtain a feature tensor by fusing the first output tensor and the second output tensor.

[0050] like Figure 3 As shown, a neural network is established in advance, which includes an angle temporal feature attention module and an entropy temporal feature attention module; the angle temporal feature attention module consists of an LSTM layer, a temporal attention layer 1, a fully connected layer 1 and a tanh1 layer, and the entropy temporal feature attention module consists of an LSTM layer, a temporal attention layer 2, a fully connected layer 2 and a tanh2 layer, and LSTM is Long Short-Term Memory.

[0051] like Figure 3 As shown, the angle timing features of the arm and the entropy timing features of the head bone points are input into the pre-established neural network, that is, the angle timing features of the arm and the entropy timing features of the head bone points are respectively input into the angle timing feature attention module and the entropy timing feature attention module, and the first output tensor and the second output tensor containing the abnormal frame information are output. The first output tensor and the second output tensor are fused into a feature tensor using the Concat function. The Concat function merges the number of channels, that is, the number of features (channels) describing the image itself is increased, while the information under each feature is not increased. After Concat fusion:

[0052] F fusion =[F out_Q ,F out_K ]

[0053] Among them, F fusion represents the feature tensor output of the fusion layer of the continuous k-frame skeleton sequence, F out_Q With F out_K They represent the first output tensor of the angle temporal feature attention module and the second output tensor of the entropy temporal feature attention module respectively.

[0054] S4. Input the feature tensor into a classifier consisting of a fully connected layer and a Sigmoid function for classification. The classifier has been trained in advance and outputs the probability of a person falling or not falling in the abnormal frame. The one with the highest probability is the detection result.

[0055] The classifier consisting of a fully connected layer and a Sigmoid function is trained in advance. The training process uses a binary cross entropy loss function, which is:

[0056]

[0057] Among them, Z is the binary cross entropy loss, N is the number of samples, a represents the counter, a=1 means counting from the first sample, is the label of the sample, the positive class is 1, the negative class is 0, y a is the probability of the prediction being positive.

[0058] like Figure 3 As shown in the figure, the above feature tensor is input into a pre-trained classifier composed of a fully connected layer and a Sigmoid function for classification, and the probabilities of human falls and non-falls in abnormal frames are output. The one with a higher probability is the detection result, that is, if the probability of falling is high, the detection result is falling, and if the probability of not falling is high, the detection result is non-falling.

[0059] Example 2

[0060] like Figure 2 As shown, an embodiment of the present invention provides a human fall detection method based on time series features, which includes a training phase and a testing phase.

[0061] The training phase includes:

[0062] Collect the data set and preprocess the images, that is, convert the RGB channel images into HSL channel images to solve the impact of unstable lighting on the optical flow field. The conversion is performed using the following formula:

[0063]

[0064]

[0065]

[0066] Where H, S, and L correspond to hue, saturation, and lightness, respectively; R, G, and B correspond to the brightness of red, green, and blue, respectively; arccos represents the inverse cosine function; and min returns the minimum value in the given parameter table.

[0067] The calcOpticalFlowFarneBack operator in OpenCV is used to calculate the change of the optical flow field between two frames. The motion area is obtained according to the change of the optical flow field to achieve dynamic human body tracking.

[0068] The OpenPose algorithm is used to obtain the coordinates of the human arm bone points and the head bone points and extract the angle timing features of the arm bone points and the entropy timing features of the head bone points. First, the coordinates of the human arm bone points and the head bone points are collected by the OpenPose algorithm, and then the angle features of the upper and lower arms are calculated by the arm bone point coordinates. The angle features of the arms in each frame of the video sample are compared with the angle features of the arms in the previous frame to obtain the angle timing features of the arms. At the same time, the motion direction and amplitude are calculated by the coordinates of the head bone points, and then the entropy of the motion direction and the entropy of the amplitude are calculated. The two are multiplied to obtain the entropy of the head bone points. The entropy of the head bone points in each frame of the video sample is compared with the entropy of the head bone points in the previous frame to obtain the entropy timing features of the head bone points.

[0069] To facilitate the calculation of arm angle features, the arm joints are marked with A, D, and M, the lower arm is marked as AD, the upper arm is marked as DM, and the angle between the upper and lower arms is marked as O. The angle difference ΔO between the previous and next frames is calculated, and the angle threshold α is set. If ΔO exceeds α, it is judged that there is a high possibility of falling. The specific calculation steps are as follows:

[0070] 1) Calculate the angle O between the upper and lower arms:

[0071]

[0072] Among them, arccos represents the inverse cosine function, C AD Represents the distance between the A and D bone points, C DM Represents the distance between the D and M bone points.

[0073] 2) Calculate the angle difference ΔO that can describe the change in posture at time t during the fall:

[0074] ΔO=O t -O t-1

[0075] Among them, O t is the angle characteristic of the arm at time t, O t-1 is the angle characteristic of the arm at time t-1.

[0076] 3) Calculate the angular temporal feature Q of the skeleton sequence with a sequence length of T composed of r consecutive moments:

[0077] Q={ΔO r |r=1,2...T}

[0078] Wherein, ΔOr represents the angular difference between time r and time r-1.

[0079] Let the head bone points be V, calculate the entropy of V, divide the entropy of V of each frame by the entropy of V of the previous frame to get the ratio, set the ratio threshold β, if the ratio exceeds β, it is judged that the possibility of falling is high. The specific steps are as follows:

[0080] 1) Assume that the coordinates of V at time t are (x t ,y t ), the coordinate of V at time t+1 is (x t+1 ,y t+1 ), d represents the distance between the position of V at time t and the position of V at time t+1, that is, the amplitude, and θ represents the angle between the position of V at time t and the position of V at time t+1, that is, the direction of movement of V.

[0081] Calculate the magnitude d of V:

[0082]

[0083] Calculate the direction of motion θ of V:

[0084] θ=atan2(y t+1 -y t , x t+1 -x t )

[0085] atan2 means calculating the arc tangent value of the horizontal and vertical coordinate points.

[0086] 2) Calculate the direction and amplitude of V's movement and use a direction-amplitude histogram to describe human behavior. The histogram's horizontal axis is divided into 12 direction intervals with an interval size of 30°, and the vertical axis is the statistical value of the amplitude in each interval. First, calculate the probability of the direction and amplitude in each interval.

[0087] The probability P(s) of the direction of the i-th interval i )for:

[0088]

[0089] Among them, i is the counter, s i is the total number of directions in the i-direction interval, S direction is the sum of the direction times of the direction-magnitude histogram.

[0090] The probability P(h i )for:

[0091]

[0092] Among them, i is the counter, h i is the amplitude of the i-th interval, S hypotenuse is the sum of the magnitudes of the direction-magnitude histogram.

[0093] 3) Calculate the entropy of the direction of motion and the entropy of the amplitude:

[0094] Entropy E of the direction of motion s :

[0095]

[0096] Where i is a counter.

[0097] The entropy of the amplitude E h :

[0098]

[0099] Where i is a counter.

[0100] The entropy of the motion direction and the entropy of the amplitude are combined into the entropy W of V:

[0101] W=E s ×E h

[0102] 4) The ratio of the entropy of V in the next frame to the entropy of V in the previous frame is recorded as F, that is:

[0103]

[0104] Among them, W t With Wt-1 Represent the entropy of V at time t and time t-1 respectively.

[0105] 5) Calculate the angular temporal feature K of the skeleton sequence with a sequence length of U composed of e consecutive moments:

[0106] K={F e |e=1,2...U}

[0107] Among them, F e It represents the ratio of the entropy of V at time e to that at time e-1.

[0108] like Figure 3 As shown, the angle timing features of the arm and the entropy timing features of V are respectively input into the angle timing feature attention module and the entropy timing feature attention module to obtain the first output tensor and the second output tensor containing the abnormal frame information; the above-mentioned angle timing feature attention module is composed of an LSTM layer, a temporal attention layer 1, a fully connected layer 1 and a tanh1 layer, and the entropy timing feature attention module is composed of an LSTM layer, a temporal attention layer 2, a fully connected layer 2 and a tanh2 layer.

[0109] like Figure 3 As shown in the figure, the first output tensor and the second output tensor are fused using Concat to more comprehensively detect the fall process. Concat merges the number of channels, that is, the number of features (channels) describing the image itself increases, while the information under each feature does not increase. After Concat fusion:

[0110] F fusion =[F out_Q , F out_K ]

[0111] Among them, F fusion represents the feature tensor output of the fusion layer of the continuous k-frame skeleton sequence, F out_Q With F out_K They represent the first output tensor of the angle temporal feature attention module and the second output tensor of the entropy temporal feature attention module respectively.

[0112] After the above feature tensor passes through the fully connected layer, the activation function Sigmoid is used to determine whether it is a fall. The binary cross entropy loss function is used during training, and the function is:

[0113]

[0114] Among them, Z is the binary cross entropy loss, N is the number of samples, a represents the counter, a=1 means counting from the first sample, is the label of the sample, the positive class is 1, the negative class is 0, y ais the probability of the prediction being positive.

[0115] The testing phase includes:

[0116] In home monitoring, a surveillance video is selected as a video sample. The video sample is first preprocessed, and the image of the RGB channel in the video sample is converted into an image of the HSL channel. Then, the human body movement area is obtained through the change of the optical flow field. The frame-by-frame decomposition and comparison are performed based on the video sample to extract the angle timing features of the arm and the temporal features of the entropy of the head bone points; the angle timing features of the arm and the temporal features of the entropy of the head bone points are input into a pre-established neural network, and a first output tensor and a second output tensor containing abnormal frame information are output; the first output tensor and the second output tensor are fused to obtain a feature tensor, and the feature tensor is input into a classifier composed of a fully connected layer and a Sigmoid function for classification. The classifier has been trained in advance and outputs the probability of a human body falling or not falling in the abnormal frame, where the one with a higher probability is the detection result.

[0117] Example 3

[0118] An embodiment of the present invention provides a human fall detection system based on time series features, comprising:

[0119] Input module: used to obtain video samples;

[0120] Temporal feature extraction module: used to decompose and compare video samples frame by frame, extracting the temporal features of arm angles and the temporal features of the entropy of head bone points;

[0121] Feature tensor output module: used to input the time series features of the arm angle and the time series features of the entropy of the head bone points into the pre-established neural network, output the first output tensor and the second output tensor containing the abnormal frame information, and fuse the first output tensor and the second output tensor to obtain the feature tensor;

[0122] Detection result classification module: The feature tensor is input into a classifier consisting of a fully connected layer and a Sigmoid function for classification. The classifier has been trained in advance and outputs the probability of a person falling or not falling in the abnormal frame. The one with the highest probability is the detection result.

[0123] Those skilled in the art will appreciate that the embodiments of the present application can be provided as methods, systems, or computer program products. Therefore, the present application can adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application can adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to magnetic disk storage, CD-ROM, optical storage, etc.) that contain computer-usable program code.

[0124] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the steps in the process. Figure 1 a process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0125] These computer program instructions may also be stored in a computer readable memory that can direct a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 a process or multiple processes and / or boxes Figure 1 The function specified in one or more boxes.

[0126] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operational steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing the instructions executed on the computer or other programmable device for implementing the process. Figure 1 a process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0127] The above is only a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the technical principles of the present invention. These improvements and modifications should also be regarded as the scope of protection of the present invention.

Claims

1. A human fall detection method based on time series features, characterized in that: include: Get video samples; Decompose and compare the video samples frame by frame to extract the temporal features of the arm angle and the temporal features of the entropy of the head bone points; Input the time series features of the arm angle and the time series features of the entropy of the head bone points into a pre-established neural network to obtain a first output tensor and a second output tensor containing abnormal frame information. After fusing the first output tensor and the second output tensor, a feature tensor is obtained. The feature tensor is input into a classifier composed of a fully connected layer and a Sigmoid function for classification. The classifier has been pre-trained and outputs the probability of a person falling or not falling in the abnormal frame. The one with the higher probability is the detection result; Decompose and compare the video samples frame by frame, including: The human body motion area is obtained through the change of the optical flow field to achieve human body dynamic tracking. The coordinates of the arm bone points in the video sample are obtained through the OpenPose algorithm. The angle features of the arm are calculated based on the coordinates of the arm bone points. The angle features of the arm are the angle features of the upper and lower arms. The angle features of the arm in each frame of the video sample are compared with the previous frame to obtain the angle time series features of the arm; The human motion area is obtained through changes in the optical flow field to achieve dynamic human tracking. The OpenPose algorithm is used to obtain the coordinates of the head skeleton points in the video sample, and the motion direction and amplitude of the head skeleton points are calculated. The entropy of the motion direction and the entropy of the amplitude are then calculated. The entropy of the motion direction and the entropy of the amplitude are multiplied to obtain the entropy of the head skeleton points. The entropy of the head skeleton points in each frame of the video sample is compared with the previous frame to obtain the temporal characteristics of the entropy of the head skeleton points. The feature tensor is obtained by the following method: The angle timing features of the arm and the entropy timing features of the head bone points are input into a pre-established neural network, which includes an LSTM-based angle timing feature attention module and an entropy timing feature attention module. The angle timing features of the arm and the entropy timing features of the head bone points are respectively input into the angle timing feature attention module and the entropy timing feature attention module to obtain a first output tensor and a second output tensor containing abnormal frame information. The first output tensor and the second output tensor are fused into a feature tensor using the Concat function.

2. The method for detecting a human fall based on time series characteristics according to claim 1, wherein: It also includes the steps of preprocessing the video samples: The video sample The channel image is converted into The image of the channel is converted to solve the impact of unstable illumination on the optical flow field. The conversion is performed using the following formula: ； ； ； in, Corresponding to hue, saturation and lightness respectively, Corresponding to the brightness of red, green and blue respectively, represents the inverse cosine function, Returns the minimum value in the given parameter table.

3. The method for detecting human falls based on time series characteristics according to claim 1, wherein: The classifier is pre-trained using a binary cross entropy loss function, which is: ； in, is the binary cross entropy loss, is the number of samples, Represents a counter, Indicates that counting starts from the first sample. is the label of the sample, the positive class is 1, the negative class is 0, is the probability of the prediction being positive.

4. The method for detecting human falls based on time series characteristics according to claim 1, wherein: The calcOpticalFlowFarneBack operator in OpenCV is used to calculate the change of the optical flow field between two frames. The motion area is obtained according to the change of the optical flow field to achieve dynamic human body tracking.

5. A human fall detection system based on time series characteristics, characterized in that: include: Input module: used to obtain video samples; Temporal feature extraction module: used to decompose and compare video samples frame by frame, extracting the temporal features of arm angles and the temporal features of the entropy of head bone points; Feature tensor output module: used to input the time series features of the arm angle and the time series features of the entropy of the head bone points into the pre-established neural network, obtain the first output tensor and the second output tensor containing the abnormal frame information, and obtain the feature tensor after fusing the first output tensor and the second output tensor; Detection result classification module: This module is used to input the feature tensor into a classifier composed of a fully connected layer and a Sigmoid function for classification. The classifier has been pre-trained and outputs the probability of a person falling or not falling in the abnormal frame. The one with the highest probability is the detection result. The video samples are decomposed and compared frame by frame, including: The human body motion area is obtained through the change of the optical flow field to achieve human body dynamic tracking. The coordinates of the arm bone points in the video sample are obtained through the OpenPose algorithm. The angle features of the arm are calculated based on the coordinates of the arm bone points. The angle features of the arm are the angle features of the upper and lower arms. The angle features of the arm in each frame of the video sample are compared with the previous frame to obtain the angle time series features of the arm; The human motion area is obtained through changes in the optical flow field to achieve dynamic human tracking. The OpenPose algorithm is used to obtain the coordinates of the head skeleton points in the video sample, and the motion direction and amplitude of the head skeleton points are calculated. The entropy of the motion direction and the entropy of the amplitude are then calculated. The entropy of the motion direction and the entropy of the amplitude are multiplied to obtain the entropy of the head skeleton points. The entropy of the head skeleton points in each frame of the video sample is compared with the previous frame to obtain the temporal characteristics of the entropy of the head skeleton points. The feature tensor is obtained by the following method: The angle timing features of the arm and the entropy timing features of the head bone points are input into a pre-established neural network, which includes an LSTM-based angle timing feature attention module and an entropy timing feature attention module. The angle timing features of the arm and the entropy timing features of the head bone points are respectively input into the angle timing feature attention module and the entropy timing feature attention module to obtain a first output tensor and a second output tensor containing abnormal frame information. The first output tensor and the second output tensor are fused into a feature tensor using the Concat function.

Images