A multi-feature fusion human fall detection method and system

Abstract

The invention discloses a multi-feature fusion human fall detection method and system, which belongs to the technical field of image processing and pattern recognition. The method comprises: obtaining a sample video; segmenting the sample video frame by frame, extracting skeletal joints and their spatial distribution features and time features therefrom; fusing the spatial distribution features and the time features to obtain a feature tensor; inputting the feature tensor into a pre-trained STM-GRU to obtain the confidence of the fall, if the confidence is greater than a preset confidence threshold, the detection result is a fall, otherwise the detection result is a non-fall; the problems in the prior art such as the inability to realize fall detection due to forgetting or not liking to wear a wearable device, and the complex system and high cost are solved, and the detection accuracy is improved.

Description

Technical Field

[0001] The present invention relates to a multi-feature fusion human body fall detection method and system, belonging to the technical field of image processing and pattern recognition. Background Art

[0002] The field of artificial intelligence has developed rapidly, and more and more smart products have entered people's lives. Although there are many smart products related to falls, most of them are complicated to wear and have poor effects. Fall recognition is one of the most important factors in home-based elderly care and plays a key role. Therefore, the field of human-computer interaction that can sense falls and respond accordingly is in urgent need of development. However, how to improve the accuracy of computer fall recognition is still a major problem.

[0003] At present, human fall detection mainly includes detection based on wearable devices and detection based on external environment sensors. The detection method based on wearable devices usually embeds sensors into portable items such as bracelets and belts to collect human motion data, and then completes fall detection through threshold method, SVM, etc. Wearable devices need to be worn by the elderly at all times. For some elderly people, especially those with memory loss, they may forget them. Some other elderly people may resist such devices due to physical comfort and do not like to wear them. The detection method based on external environment sensors usually uses the changes in some physical quantities in the environment such as pressure, vibration, and infrared heat sources caused by human motion to identify fall behavior. This method requires the deployment of sensors in the environment and is affected by the surrounding environmental noise. The system is relatively complex and costly. Summary of the invention

[0004] The purpose of the present invention is to provide a multi-feature fusion human fall detection method and system to solve the problems in the prior art such as the inability to realize fall detection due to forgetting or not liking to wear wearable devices, and the complex and high cost of the system, so as to improve the accuracy of detection.

[0005] To achieve the above objectives, the present invention is implemented by adopting the following technical solutions:

[0006] In a first aspect, the present invention provides a multi-feature fusion human fall detection method, comprising:

[0007] Get sample video;

[0008] Segment the sample video frame by frame and extract the skeleton joint points and their spatial distribution features and temporal features;

[0009] The spatial distribution features and time features are fused to obtain the feature tensor;

[0010] The feature tensor is input into the pre-trained STM-GRU to obtain the confidence of the fall. If the confidence is greater than the preset confidence threshold, the detection result is a fall, otherwise the detection result is a non-fall.

[0011] In combination with the first aspect, further, the spatial distribution feature includes the accumulation of Euclidean distances between normalized coordinate points of the skeletal joint points and the reference points; the normalized coordinate points are obtained by:

[0012] Compute the normalized coordinate matrix:

[0013]

[0014] Among them, P std represents the normalized coordinate matrix, P represents a numpy matrix of shape (K, D), K represents the number of bone joints, D represents the coordinate dimension, P .axis_max Indicates the maximum value of the coordinate, P .axis_min Represents the minimum value of the coordinates; each element in the normalized coordinate matrix is ​​the normalized coordinate point of the bone joint point;

[0015] The Euclidean distance between two points is calculated using the following formula:

[0016]

[0017] Where E represents the Euclidean distance, x1 and x2 are the horizontal coordinates of the two points, and y1 and y2 are the vertical coordinates of the two points;

[0018] After calculating the Euclidean distance between the normalized coordinate point of each skeletal joint point and the reference point, each Euclidean distance is added together to obtain the cumulative Euclidean distance between the normalized coordinate point of the skeletal joint point and the reference point.

[0019] In combination with the first aspect, further, the spatial distribution feature includes a length-to-width ratio of a human body outline rectangle, and the length and width of the human body outline rectangle are calculated by the following method:

[0020] Obtaining a head region bone joint point set and a foot region bone joint point set from the bone joint points, and selecting a point from each of the two sets, and using the difference in ordinates between the two selected points to represent the length of the human body contour rectangle;

[0021] A left-side region bone joint point set and a right-side region bone joint point set are obtained from the bone joint points, and one point is selected from each of the two sets. The width of the human body contour rectangle is represented by the difference in the horizontal coordinates between the two selected points.

[0022] In combination with the first aspect, further, the spatial distribution characteristics include key angles, and the key angles include a first angle between a thigh vector and a calf vector, a second angle between a back vector and a thigh vector, and a third angle between a back vector and a horizontal vector.

[0023] In combination with the first aspect, further, the time feature includes a head descent speed, and the head descent speed is calculated by the following formula:

[0024]

[0025] Among them, Vh represents the falling speed of the human head, Represents the coordinates of the head bone joint point p15 at time t1, Represents the coordinates of the head bone joint point p15 at time t2.

[0026] In combination with the first aspect, further, the time feature includes a waist descending speed, and the waist descending speed is calculated by the following formula:

[0027]

[0028] Among them, Vw represents the speed of the waist descent of the human body, Indicates the coordinates of waist bone joint point p8 at time t1, Represents the coordinates of waist bone joint point p8 at time t2.

[0029] In combination with the first aspect, further, the skeletal joint points are obtained by the following method:

[0030] After the video samples are segmented frame by frame, the skeleton joint points are extracted using the openpose algorithm.

[0031] In combination with the first aspect, further, the feature tensor is obtained by the following method:

[0032] The spatial distribution features and time features are fused through the Concat function to obtain the feature tensor.

[0033] In combination with the first aspect, further, the method for pre-training the STM-GRU includes:

[0034] The spatial distribution features and time features are input into STM-GRU for training. During the training process, the update gate is used inside STM-GRU to replace the input gate and forget gate to control the cell state;

[0035] The confidence of the fall is obtained by the following method:

[0036] Each spatial distribution feature and time feature is multiplied by their respective weighting factors and then summed up to obtain the confidence of the fall.

[0037] In a second aspect, the present invention further provides a multi-feature fusion human fall detection system, comprising:

[0038] Data acquisition module: used to obtain sample videos;

[0039] Feature extraction module: used to segment the sample video frame by frame and extract the skeleton joint points and their spatial distribution features and temporal features;

[0040] Feature fusion module: used to fuse spatial distribution features and temporal features to obtain feature tensors;

[0041] Detection module: used to input the feature tensor into the pre-trained STM-GRU to obtain the confidence of the fall. If the confidence is greater than the preset confidence threshold, the detection result is a fall, otherwise the detection result is a non-fall.

[0042] Compared with the prior art, the beneficial effects achieved by the present invention are:

[0043] The present invention provides a multi-feature fusion human fall detection method and system, which extracts the spatial distribution characteristics and time characteristics of skeletal joints, fuses the two characteristics, and inputs them into a pre-trained STM-GRU to complete the accurate recognition and detection of "falls". The coordinates of the skeletal joints are normalized to obtain normalized coordinates, and a spatial distribution feature is calculated by the normalized coordinates, so that the result is not affected by the change of the target position in the sample video, thereby improving the accuracy of detection. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] Figure 1 is a flow chart of a multi-feature fusion human fall detection method provided by an embodiment of the present invention;

[0045] Figure 2 It is a human body posture diagram provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0046] The present invention will be further described below in conjunction with the accompanying drawings. The following embodiments are only used to more clearly illustrate the technical solutions of the present invention and are not intended to limit the protection scope of the present invention.

[0047] Example 1

[0048] like Figure 1 As shown, a multi-feature fusion human fall detection method provided by an embodiment of the present invention includes:

[0049] S1. Obtain sample video.

[0050] First, obtain the sample video to be detected.

[0051] S2. Segment the sample video frame by frame to extract the skeleton joints and their spatial distribution features and temporal features.

[0052] The sample video is segmented into each frame, and the obtained color image is used to extract the skeleton joint points using the openpose algorithm.

[0053] Each frame in the sample video corresponds to each static posture of the human body falling. The sample video can be regarded as a skeleton point set composed of a series of skeletal joint coordinates. All samples output by Openpose are converted into skeletal joints of the human body. The direct output of Openpose is 25 skeletal joints. These skeletal joints need to be preprocessed to obtain the skeletal joints that can be input into the neural network. Figure 2 The human body posture diagram shown.

[0054] In the preprocessing stage, the skeletal joint data will be cleaned up. For invalid samples where the human body is not detected and the target skeletal joint data is missing or the valid skeletal joint data is below a certain threshold, it is sufficient to discard this frame of data. Because a sample video corresponds to a sufficient number of frames, the amount of skeletal joint data is large enough, and the coverage and importance of invalid samples are very low, deletion has basically no effect on the implementation of the solution of the present invention.

[0055] When individual skeletal joints are occluded, such as the feet or the head, they are replaced by surrounding skeletal joints or the geometric centers of surrounding skeletal joints. For redundant noise samples in which a single target is mistakenly detected as multiple targets, the valid skeletal joints of each skeleton can be counted according to the confidence of the skeletal joints in the sample. The skeletal joints with confidence greater than a certain threshold are valid skeletal joints. The data of the skeleton point set with a large number of valid skeletal joints is selected as the true target of the sample. The preprocessing stage will prepare the "cleaned" skeletal joint data for the next stage.

[0056] To process the bone joint data, we need to extract the spatial distribution features first. In order to extract the spatial distribution features of each skeletal joint in a static posture in each frame of the image, the spatial distribution features at time t should include the relative position features F1 between skeletal joints, the aspect ratio features F2, and the angle features F3 between limbs. The spatial distribution features can be expressed as follows:

[0057]

[0058] Among them, R represents a real number set, I represents the number of convolution channels, K represents the skeleton joint point, and s represents the skeleton joint point label number.

[0059] Each frame in the sample video corresponds to each static posture of the human body falling. The sample video can be regarded as a skeleton point set composed of a series of skeletal joint point coordinates. Considering that the human skeleton is an overall framework, there is a sequence and position distribution relationship between different skeletal joint points, which also contains hidden information. These structural information cannot be ignored.

[0060] The relative position relationship between skeletal joints can distinguish the intra-class differences of falls among different individuals. Compared with the unrelated original skeletal joint coordinates, it contains the spatial distribution characteristics of each skeletal joint in a static posture of the human body, which is physically expressed as the Euclidean distance between the skeletal joint coordinates.

[0061] Considering the scaling problem of skeleton size in the same action recognition under different targets and the uncertain position of the target in the video screen, there will be a big difference when the spatial position is directly calculated from the Euclidean distance of the skeleton joint point coordinates. When calculating the Euclidean distance, the skeleton joint point coordinates of each group are first normalized to obtain the normalized coordinate matrix. The formula is as follows:

[0062]

[0063] Among them, P std represents the normalized coordinate matrix, P represents a numpy matrix of shape (K, D), K represents the number of bone joints, D represents the coordinate dimension, P .axis_max Indicates the maximum value of the coordinate, P .axis_min Represents the minimum value of the coordinates; each element in the normalized coordinate matrix is ​​the normalized coordinate point of the bone joint point.

[0064] P scaled =P std (max-min)+mmin

[0065] Among them, P scaled Represents the normalized coordinates after scaling, max and min represent the maximum and minimum values ​​of a given scaling range, and in this embodiment are taken as 1 and 0 respectively, with the purpose of normalizing the coordinates of the bone joint points to the range of 0 to 1 according to the dimension.

[0066] Calculate the Euclidean distance between the normalized coordinates of the bone joint point and the reference point. Figure 2 The 25 skeletal joint points in are denoted as p = {p0, p1, p2, ..., p24}, and the coordinates of the skeletal joint point pi are denoted as (xi, yi). In this embodiment, Figure 2 Point p8 in the figure is used as the reference point, and the Euclidean distances E from points p3, p4, p6, p7, p11, p14 to p8 are calculated respectively. The change characteristics from these six points to the center point are the most obvious during the fall. The Euclidean distance formula between two points (x1, y1) and (x2, y2) on the plane is calculated as follows:

[0067]

[0068] When an elderly person falls, his body will curl up in most cases. The Euclidean distances from points p3, p4, p6, p7, p11, p14 to p8 will be smaller than when he walks upright normally. The Euclidean distances of these points and point p8 are accumulated:

[0069] E sum =∑{E pi-p8 |i=3,4,6,7,11,14}

[0070] Among them, E pi-p8 Represents the Euclidean distance from point pi to point p8.

[0071] The skeleton aspect ratio is very meaningful for the recognition of falling actions. As the body goes from upright to falling, the skeleton aspect ratio changes significantly. Considering the camera height, the change process of the skeleton aspect ratio in the captured video is not the same. When the camera is high, the skeleton aspect ratio gradually increases from a value close to 1 or slightly greater than 1 to a value much greater than 1; when the camera is low, the skeleton aspect ratio gradually decreases from a value much greater than 1 to a value close to 1.

[0072] Calculate the length-to-width ratio of the human body contour rectangle, obtain the set of bone joint points in the head area and the set of bone joint points in the foot area from the bone joint points, select a point from each of the two sets, and use the difference between the vertical coordinates of the two selected points to represent the length of the human body contour rectangle; obtain the set of bone joint points in the left area and the set of bone joint points in the right area from the bone joint points, select a point from each of the two sets, and use the difference between the horizontal coordinates of the two selected points to represent the width of the human body contour rectangle.

[0073] When a person walks upright normally, the vertical direction of the human body outline rectangle is larger than the horizontal direction, but if the person lies down, the horizontal direction is larger than the vertical direction; it is more appropriate to use the human shoulder information to represent the width of the outline, and the distance between the head and the feet to represent the height of the outline; therefore, in this embodiment, the selection Figure 2 The coordinates of the above four points are expressed as (x2, y2), (x5, y5), (x15, y15), (x22, y22), and the aspect ratio H of the human body outline rectangle can be calculated by the following formula:

[0074]

[0075] Calculate the key angles, which include a first angle Ang1 between a thigh vector and a calf vector, a second angle Ang2 between a back vector and a thigh vector, and a third angle Ang3 between a back vector and a horizontal vector.

[0076] In real scenes, the direction in which a person falls cannot be determined, but after summarizing the process of falling, we can conclude that the bending of the thigh will cause the first angle Ang1 between the thigh vector and the calf vector to first increase and then decrease, and the change is more obvious; when the human body is in an upright state, the back and legs are basically in a straight line, and when a fall occurs, the second angle Ang2 between the back vector and the thigh vector gradually increases and then decreases; for convenience, the coordinate information of the left leg is uniformly selected as the basis for calculating Ang1 and Ang2.

[0077] We denote the back vector C back , the left and right thigh vectors are C lbleg ,C rbleg , the left and right calf vectors are C lsleg ,C rsleg , the horizontal vector is C horizon , the vertical vector is C vertical , they are respectively expressed as follows:

[0078] C back =(p1, p8)

[0079] C horizon =(1,0)

[0080] C vertical =(0,1)

[0081] C lbleg =(p12, p13)

[0082] C rbleg =(p9, p10)

[0083] C lsleg =(p13, p14)

[0084] C rsleg =(p10, p11)

[0085] Next, we can calculate the key angle mentioned above, the formula is as follows:

[0086]

[0087]

[0088]

[0089] In addition to extracting the above-mentioned spatial feature information, we also need to extract temporal feature information. The bone joint point data corresponding to multiple consecutive frames in the video constitute a skeleton action sequence. How to select a feature to describe the skeleton sequence data that changes dynamically over time is also a problem to be solved by the solution of the present invention.

[0090] The corresponding bone joint point data of multiple consecutive frames in the video constitute a skeleton action sequence. As time changes, the spatial position change information of the same bone joint point constitutes the motion trajectory of the bone joint point, and the descent speed of the human head and waist is calculated respectively.

[0091] Select Figure 2 The head bone joint point p15 in the image is used to calculate the head descent speed, which is calculated using the following formula:

[0092]

[0093] Among them, Vh represents the falling speed of the human head, represents the coordinates of the head bone point p15 at time t1, Represents the coordinates of the head bone joint point p15 at time t2.

[0094] Select Figure 2 The waist bone joint point p8 in the figure is used to calculate the head descending speed, which is calculated by the following formula:

[0095]

[0096] Among them, Vw represents the speed of the waist descent of the human body, Indicates the coordinates of waist bone joint point p8 at time t1, Represents the coordinates of waist bone joint point p8 at time t2.

[0097] Because in the dynamic process of human body descent, the bone joint point information is easily lost. If the p8 point is lost, you can choose P9 or P12 on the waist as the waist reference point. Similarly, you can choose P16 on the head.

[0098] S3. Fuse the spatial distribution features and the temporal features to obtain the feature tensor.

[0099] The spatial distribution features and time features are fused through the Concat function to obtain the feature tensor.

[0100] Information fusion can comprehensively describe the temporal and spatial distribution characteristics of action skeleton sequences. Common fusion methods include Max, Mean and Concat. The Max method compares each element and takes the larger value and discards the smaller value. The spatial feature and the temporal feature have different dimensions. It is unreasonable to discard the smaller value and some information will be lost. The Mean method calculates the mean of the spatiotemporal tensor element by element, which is equivalent to multiplying the spatiotemporal tensor by a weight coefficient of 0.5 and then adding them together. It requires that the shapes of the spatiotemporal tensors must be the same, but the shapes of the spatiotemporal tensors in this model are not strictly the same, so this method is not suitable. In comparative experiments, the algorithm in this paper is most suitable for the Concat method. The Concat fusion method has relatively stable performance under large and small data volumes, and has strong robustness in the case of multi-channel feature fusion. It can also combine and complement the information output under different features to provide multi-faceted detailed data for subsequent classification modules.

[0101] After Concat fusion, it is expressed as:

[0102] F fusion =[F out_s ,F out_t ]

[0103] where F out_s, F out_t Represent the spatial distribution characteristics and temporal characteristics respectively, F fusion The feature tensor representing the fusion layer of a sequence of consecutive T-frame skeletons.

[0104] S4. Input the feature tensor into the pre-trained STM-GRU to obtain the confidence of the fall. If the confidence is greater than the preset confidence threshold, the detection result is a fall, otherwise the detection result is a non-fall.

[0105] After feature fusion is completed, a binary classification method is selected to distinguish between falls and non-falls. A Dropout layer is added to the STM-GRU so that the model randomly and temporarily discards some neurons with a certain probability during the training process to avoid overfitting of the model. The Adam optimizer is used to train the optimal model, which is suitable for solving problems with large-scale data or parameters, with high computational efficiency and low memory requirements. The loss function used in logistic regression is the binary cross entropy loss function Sigmoid, and the formula is as follows:

[0106]

[0107] Where M is the number of samples, yi is the label of the sample feature value xi, that is, the real sample, the positive class is 1, the negative class is 0, f(x(i)) is the probability of predicting to be positive, where f is the sigmoid function.

[0108] STM-GRU stands for Short Term Memory-Gate Recurrent Unit, that is, short-term memory neural network-gated recurrent unit.

[0109] The accuracy rate (Accuracy) and the missed alarm rate (MA) are selected as the evaluation criteria of the fall model. The specific calculation formula is as follows:

[0110]

[0111]

[0112] The falling action is set as the positive sample, and the other actions are set as negative samples. In the formula, the true positive sample TP (True Positive) represents the falling sample predicted by the model as a fall; the true negative sample TN (True Negative) represents the non-fall sample predicted by the model as a non-fall; the false negative sample FN (False Negative) represents the falling sample predicted by the model as a non-fall; P and N represent positive and negative samples, respectively.

[0113] The spatial distribution features and time features in step S2 are weighted and calculated, and the weighted factor ratio is: the sum of the Euclidean distances between the normalized coordinate points of the skeletal joint points and the reference points is α0, the time feature is α1, the aspect ratio of the human body contour rectangle is α2, the first angle, the second angle, and the third angle are α3, α4, and α5 respectively, and α0+α1+α2+α3+α4+α5=1.

[0114] In this embodiment, the threshold Ep of the Euclidean distance from the normalized coordinate point to the reference point p8 is set to 0.3, the threshold Vp of the human body descent speed is set to 1.5 m / s, the threshold Hp of the human body contour rectangle height ratio is set to 2:3, the threshold of the first angle is set to 90 degrees, the threshold of the second angle is set to 100 degrees, and the threshold of the third angle is set to 60 degrees.

[0115] In the pre-built STM-GRU, the confidence of the fall is calculated by the following formula:

[0116]

[0117] Among them, Ang P1 Indicates the threshold of the first angle, Ang P2 Indicates the threshold of the second angle, Ang P3 Indicates the threshold of the third angle.

[0118] In this embodiment, the values ​​of α0 to α5 are set to 0.15, 0.35, 0.2, 0.1, 0.1, and 0.1, respectively.

[0119] If the confidence level is greater than a preset confidence threshold, the detection result is a fall, otherwise the detection result is a non-fall; in this embodiment, the confidence threshold is 0.75.

[0120] This embodiment also includes a training process, in which spatial distribution features and time features are input into STM-GRU for training. Action recognition based on skeleton sequence data is a sequence prediction problem. Traditional recurrent neural network RNN ​​(Recurrent Neural Network) and its variants are suitable for processing sequence data. Although the long short-term memory neural network LSTM (Long Short Term Memory) proposed in 1997 solves the problem of gradient explosion or disappearance of traditional RNN, the model training is slow due to the complex structure of the model itself. The gated recurrent unit GRU (GateRecurrent Unit) proposed in 2014 improves LSTM. The update gate is used inside the unit instead of the input gate and the forget gate to control the cell state. The number of parameters is less than that of LSTM. The advantage of this method is that the amount of calculation can be reduced, the training speed will be faster, and the model expression ability is not weaker than LSTM.

[0121] In the STM-GRU described in this embodiment, the number of GRU hidden units is 128, the number of neurons in the fully connected layer of the spatiotemporal attention module is 64, the number of neurons in the first fully connected layer of the classification module is 32, and the number of neurons in the second fully connected layer is 4, the number of action categories.

[0122] After experiments, when the confidence threshold is set to 0.75, the accuracy can reach 97.38%, and the false alarm rate MA is 2.69%, which is the best value in this experimental test. The above data can show the effectiveness of the solution of the present invention.

[0123] Example 2

[0124] An embodiment of the present invention provides a multi-feature fusion human fall detection system, comprising:

[0125] Data acquisition module: used to obtain sample videos;

[0126] Feature extraction module: used to segment the sample video frame by frame and extract the skeleton joint points and their spatial distribution features and temporal features;

[0127] Feature fusion module: used to fuse spatial distribution features and temporal features to obtain feature tensors;

[0128] Detection module: used to input the feature tensor into the pre-established STM-GRU to obtain the confidence of the fall. If the confidence is greater than the preset confidence threshold, the detection result is a fall, otherwise the detection result is a non-fall.

[0129] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented in one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that include computer-usable program code.

[0130] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0131] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0132] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0133] The above is only a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the technical principles of the present invention. These improvements and modifications should also be regarded as the scope of protection of the present invention.

Claims

1. A multi-feature fusion human fall detection method, characterized in that: include: Get sample video; Segment the sample video frame by frame and extract the skeleton joint points and their spatial distribution features and temporal features; The spatial distribution features and time features are fused to obtain the feature tensor; The feature tensor is input into the pre-trained STM-GRU to obtain the confidence of the fall. If the confidence is greater than the preset confidence threshold, the detection result is a fall, otherwise the detection result is a non-fall; The spatial distribution feature includes the accumulation of the Euclidean distance between the normalized coordinate point of the skeletal joint point and the reference point; the normalized coordinate point is obtained by: Compute the normalized coordinate matrix: Among them, P std represents the normalized coordinate matrix, P represents a numpy matrix of shape (K, D), K represents the number of bone joints, D represents the coordinate dimension, P .axis_max Indicates the maximum value of the coordinate, P .axis_min Represents the minimum value of the coordinates; each element in the normalized coordinate matrix is ​​the normalized coordinate point of the bone joint point; The Euclidean distance between two points is calculated using the following formula: Where E represents the Euclidean distance, x1 and x2 are the horizontal coordinates of the two points, and y1 and y2 are the vertical coordinates of the two points; After calculating the Euclidean distance between the normalized coordinate point of each skeletal joint point and the reference point, each Euclidean distance is added to obtain the cumulative Euclidean distance between the normalized coordinate point of the skeletal joint point and the reference point; The spatial distribution feature includes the length-to-width ratio of the human body outline rectangle, and the length and width of the human body outline rectangle are calculated by the following method: Obtaining a head region bone joint point set and a foot region bone joint point set from the bone joint points, and selecting a point from each of the two sets, and using the difference in ordinates between the two selected points to represent the length of the human body contour rectangle; Obtaining a left-side region bone joint point set and a right-side region bone joint point set from the bone joint points and selecting a point from each of the two sets, and using the difference in the horizontal coordinates between the two selected points to represent the width of the human body contour rectangle; The spatial distribution feature includes key angles, and the key angles include a first angle between a thigh vector and a calf vector, a second angle between a back vector and a thigh vector, and a third angle between a back vector and a horizontal vector; The time feature includes the head descent speed, which is calculated by the following formula: Among them, Vh represents the falling speed of the human head, Represents the coordinates of the head bone joint point p15 at time t1, Represents the coordinates of the head bone joint point p15 at time t2; The time feature includes the waist descent speed, which is calculated by the following formula: Among them, Vw represents the speed of the waist descent of the human body, Indicates the coordinates of waist bone joint point p8 at time t1, Represents the coordinates of waist bone joint point p8 at time t2.

2. A multi-feature fusion human fall detection method according to claim 1, characterized in that: The bone joint points are obtained by the following method: After the video samples are segmented frame by frame, the skeleton joint points are extracted using the openpose algorithm.

3. The multi-feature fusion human fall detection method according to claim 1, characterized in that: The feature tensor is obtained by the following method: The spatial distribution features and time features are fused through the Concat function to obtain the feature tensor.

4. The multi-feature fusion human fall detection method according to claim 1, characterized in that: The STM-GRU pre-training method includes: The spatial distribution features and time features are input into STM-GRU for training. During the training process, the update gate is used inside STM-GRU to replace the input gate and forget gate to control the cell state; The confidence of the fall is obtained by the following method: Each spatial distribution feature and time feature is multiplied by their respective weighting factors and then summed up to obtain the confidence of the fall.

5. A multi-feature fusion human fall detection system, characterized in that: include: Data acquisition module: used to obtain sample videos; Feature extraction module: used to segment the sample video frame by frame and extract the skeleton joint points and their spatial distribution features and temporal features; Feature fusion module: used to fuse spatial distribution features and temporal features to obtain feature tensors; Detection module: used to input the feature tensor into the pre-trained STM-GRU to obtain the confidence of the fall. If the confidence is greater than the preset confidence threshold, the detection result is a fall, otherwise the detection result is a non-fall; The spatial distribution feature includes the accumulation of the Euclidean distance between the normalized coordinate point of the skeletal joint point and the reference point; the normalized coordinate point is obtained by: Compute the normalized coordinate matrix: Among them, P std represents the normalized coordinate matrix, P represents a numpy matrix of shape (K, D), K represents the number of bone joints, D represents the coordinate dimension, P .axis_max Indicates the maximum value of the coordinate, P .axis_min Represents the minimum value of the coordinates; each element in the normalized coordinate matrix is ​​the normalized coordinate point of the bone joint point; The Euclidean distance between two points is calculated using the following formula: Where E represents the Euclidean distance, x1 and x2 are the horizontal coordinates of the two points, and y1 and y2 are the vertical coordinates of the two points; After calculating the Euclidean distance between the normalized coordinate point of each skeletal joint point and the reference point, each Euclidean distance is added to obtain the cumulative Euclidean distance between the normalized coordinate point of the skeletal joint point and the reference point; The spatial distribution feature includes the length-to-width ratio of the human body outline rectangle, and the length and width of the human body outline rectangle are calculated by the following method: Obtaining a head region bone joint point set and a foot region bone joint point set from the bone joint points, and selecting a point from each of the two sets, and using the difference in ordinates between the two selected points to represent the length of the human body contour rectangle; Obtaining a left-side region bone joint point set and a right-side region bone joint point set from the bone joint points and selecting a point from each of the two sets, and using the difference in the horizontal coordinates between the two selected points to represent the width of the human body contour rectangle; The spatial distribution feature includes key angles, and the key angles include a first angle between a thigh vector and a calf vector, a second angle between a back vector and a thigh vector, and a third angle between a back vector and a horizontal vector; The time feature includes the head descent speed, which is calculated by the following formula: Among them, Vh represents the falling speed of the human head, Represents the coordinates of the head bone joint point p15 at time t1, Represents the coordinates of the head bone joint point p15 at time t2; The time feature includes the waist descent speed, which is calculated by the following formula: Among them, Vw represents the speed of the waist descent of the human body, Indicates the coordinates of waist bone joint point p8 at time t1, Represents the coordinates of waist bone joint point p8 at time t2.

Images