An image-based lightweight fall detection method

Abstract

The application belongs to the technical field of image processing and machine learning, and relates to a lightweight fall detection method based on images. The method comprises the following steps: collecting and storing image sequences by using a camera; obtaining a skeleton sequence from the image sequences by using a pose estimation model; calculating the mean and variance of each frame of the skeleton, calculating the average variance and mean of the skeleton sequence, and then performing normalization processing on the skeleton sequence to obtain a normalized skeleton sequence; extracting deep features of the normalized skeleton sequence by using a space-time attention network; the space-time attention network is composed of a plurality of space-time attention encoders connected in series; the space-time attention encoder is connected in parallel with a time attention encoder and a space attention encoder; and finally, a full connection network is used to determine the fall according to the deep features. The method realizes a balanced accuracy of 97.4%, determines the fall according to the image sequence, captures the dynamic characteristics of the fall process, improves the robustness, accuracy and real-time performance of the fall detection, and reduces the detection cost.

Description

TECHNICAL FIELD

[0001] The present application belongs to the technical field of image processing and machine learning, and particularly relates to a lightweight fall detection method based on images. BACKGROUND

[0002] Population aging has become a global phenomenon. In 2020, the number of people aged 65 and over reached 760 million, accounting for 10% of the world's population. Falling is one of the main health risks faced by the elderly. Falling can lead to serious physiological consequences such as fractures and head injuries. Artificially caring for the elderly who fall consumes human resources and is costly, while an intelligent fall detection system can detect the fall of the elderly in time and alarm, minimizing the physical and mental damage caused by falling to the elderly.

[0003] Currently, fall detection technology can be divided into fall detection based on wearable sensors, fall detection based on environment, and fall detection based on images. Fall detection based on wearable sensors usually uses acceleration sensors and gyroscopes worn on the human body to detect falls. When a person falls, the movement state of each part of the body is different from usual, and the wearable sensor can capture these abnormalities, obtain features that significantly represent falls through feature extraction technology, and use decision trees, machine learning classifiers, and other methods to distinguish between normal and fall situations. Fall detection based on the environment usually uses pressure sensing devices, audio detection devices, or radars to detect the abnormal pressure of the human body on the ground, the abnormal sound emitted when falling, and the abnormal radar echo when the human body falls, thereby determining whether a person has fallen. Fall detection based on images uses RGB images, infrared thermal imaging images, depth images, and other images to detect falls. The fall detection model based on images can be divided into two steps: feature extraction and action recognition. After the target recognition model or semantic segmentation model identifies the human body in the feature extraction step, it extracts the aspect ratio, contour key points, and other hand-crafted features related to falls. The action recognition part then determines whether a person has fallen based on these features. However, these methods still have some problems and shortcomings. Fall detection based on wearable sensors requires the elderly to wear wearable sensors, which affects their lives; fall detection based on the environment requires the placement of environmental monitoring sensors in all scenes where the elderly are active, which is complex and costly; fall detection based on images requires processing a large amount of information in the image, which requires strong computing power support. SUMMARY

[0004] The purpose of the present application is to overcome the shortcomings of current image-based fall detection schemes, which consume a lot of computing power and have high costs. Therefore, a lightweight fall detection method based on images is proposed.

[0005] To achieve the above purpose, the present application adopts the following technical solutions.

[0006] The lightweight fall detection method comprises the following steps:

[0007] S1, acquiring an image sequence by using a camera and storing the image sequence;

[0008] S2, obtaining a skeleton sequence from the image sequence acquired in S1 by using a pose estimation model;

[0009] S3, calculating the mean value and the variance of each frame of the skeleton, calculating the average variance and the mean value of the skeleton sequence, and then performing normalization processing on the skeleton sequence in S2 to obtain a normalized skeleton sequence;

[0010] S4, extracting deep features from the normalized skeleton sequence obtained in S3 by using a spatio-temporal attention network;

[0011] The spatio-temporal attention network is composed of a plurality of spatio-temporal attention encoders connected in series;

[0012] The spatio-temporal attention encoder is composed of a temporal attention encoder and a spatial attention encoder connected in parallel;

[0013] S5, determining a fall by using a fully connected network according to the deep features obtained in S4.

[0014] The pose estimation model in S2 is a Lightweight OpenPose pose estimation model, and the skeleton sequence comprises N frames of skeletons and 14 key points.

[0015] The N is in the range of 20 to 60, and the 14 key points include head, neck, left and right shoulder joints, left and right elbow joints, left and right wrist joints, left and right hip joints, left and right knee joints, and left and right ankle joints.

[0016] The N is specifically 32 frames, and the 14 key points include head, neck, left and right shoulder joints, left and right elbow joints, left and right wrist joints, left and right hip joints, left and right knee joints, and left and right ankle joints.

[0017] S3 comprises the following sub-steps:

[0018] S31, calculating the x mean value and the y mean value of each frame of the skeleton sequence;

[0019] S32, calculating the x variance and the y variance of each frame of the skeleton;

[0020] S33, calculating the average variance of the skeleton sequence;

[0021] S34, calculating the x mean value and the y mean value of the skeleton sequence;

[0022] S35, normalizing the skeleton sequence in S2 to obtain a normalized skeleton sequence;

[0023] S31, specifically: x mean value of the i-th frame skeleton, denoted as μ xi ; y mean value of the i-th frame skeleton, denoted as μ yi ;

[0024] The μ xi and μ yi are calculated by the following formula (1):

[0025]

[0026] Wherein, x ij , y ij is the x coordinate and y coordinate of the j-th key point of the i-th frame skeleton in the skeleton sequence obtained in S2.

[0027] S32, specifically:

[0028] x variance of the i-th frame skeleton, denoted as y variance of the i-th frame skeleton, denoted as

[0029] The and are calculated by the following formula (2):

[0030]

[0031] S33, the average variance of the skeleton sequence, denoted as S 2 , is calculated by the following formula (3):

[0032]

[0033] S34, the x mean value of the skeleton sequence, denoted as μ x ; the y mean value of the skeleton sequence, denoted as μ y ; the μ x and μ y are calculated by the following formula (4):

[0034]

[0035] S35, the normalized skeleton sequence is calculated by formula (5):

[0036]

[0037] Wherein, x′ ij , y′ ij is the x coordinate and y coordinate of the j-th key point of the i-th frame skeleton in the normalized skeleton sequence.

[0038] The number of the plurality of structure identical spatio-temporal attention encoders S4 is K and the value range of K is 2 to 8.

[0039] The output of the spatio-temporal attention encoder is obtained by adding the output of the temporal attention encoder and the output of the spatial attention encoder S4.

[0040] The spatial attention encoder is connected in parallel with a self-attention layer and a residual connection, and then connected in series with a layer normalization and a ReLU activation function.

[0041] The temporal attention encoder is composed of a spatial attention encoder and a transpose layer added before and after the spatial attention encoder.

[0042] The full connection network S5 is composed of a full connection layer and a Softmax layer connected in series.

[0043] Beneficial effects

[0044] Compared with the existing fall detection model, the lightweight fall detection model based on images has the following beneficial effects:

[0045] 1. The fall detection model judges the fall according to the image sequence, can capture the dynamic characteristics in the fall process, and improves the robustness of the fall detection;

[0046] 2. The fall detection model extracts the correlation in time and the correlation in space of each key point in the skeleton sequence through the spatio-temporal attention encoder, comprehensively considers various information in the skeleton sequence when extracting deep features, and improves the accuracy of the fall detection;

[0047] 3. The fall detection model has small parameter quantity and low operation amount, saves more computing power compared with the existing fall detection model, and can be deployed in an edge computing device, thereby reducing the cost of the fall detection and improving the real-time performance of the fall detection. BRIEF DESCRIPTION OF DRAWINGS

[0048] Figure 1 is the model structure diagram of the lightweight fall detection method based on images;

[0049] Figure 2 is the flowchart when the model is trained and tested;

[0050] Figure 3 is the loss function value change curve diagram in the model training process of the application;

[0051] Figure 4 is the confusion matrix when the model is tested;

[0052] Figure 5is a flow chart of the present application in model deployment and practical application;

[0053] Figure 6 is an effect diagram of the present application in practical application, including fall-down discrimination (6a) and fall-down discrimination (6b). DETAILED DESCRIPTION

[0054] The lightweight fall-down detection model based on images according to the present application will be further described and explained in detail in combination with the drawings and embodiments.

[0055] Embodiment 1

[0056] The lightweight fall-down detection method, as shown in the figure, comprises the following steps: Figure 1

[0057] S1, collecting image sequences by using a camera and storing;

[0058] S2, obtaining a skeleton sequence from the image sequences collected in S1 by using a pose estimation model;

[0059] S3, calculating the mean and variance of each frame of skeleton, calculating the average variance and mean of the skeleton sequence, and then normalizing the skeleton sequence in S2 to obtain a normalized skeleton sequence;

[0060] S4, extracting deep features from the normalized skeleton sequence obtained in S3 by using a spatio-temporal attention network;

[0061] The spatio-temporal attention network is composed of a plurality of spatio-temporal attention encoders connected in series;

[0062] The spatio-temporal attention encoder is composed of a time attention encoder and a space attention encoder connected in parallel;

[0063] S5, judging fall-down according to the deep features obtained in S4 by using a fully connected network.

[0064] The pose estimation model in S2 is a Lightweight OpenPose pose estimation model, and the skeleton sequence comprises N frames of skeleton and 14 key points.

[0065] The N is 32 frames in specific implementation; the 14 key points comprise head, neck, left and right shoulder joints, left and right elbow joints, left and right wrist joints, left and right hip joints, left and right knee joints, and left and right ankle joints; each key point comprises x coordinate and y coordinate.

[0066] S3 comprises the following sub-steps:

[0067] S31, calculating the x mean and y mean of each frame of skeleton in the skeleton sequence;

[0068] ​S32, calculate the x variance and y variance of each frame skeleton;

[0069] S33, calculate the average variance of the skeleton sequence;

[0070] S34, calculate the x mean and y mean of the skeleton sequence;

[0071] S35, normalize the skeleton sequence in S2 to obtain a normalized skeleton sequence;

[0072] S31, specifically: the x mean of the i-th frame skeleton, denoted as μ xi ; the y mean of the i-th frame skeleton, denoted as μ yi ;

[0073] The μ xi and μ yi are calculated by the following formula (1):

[0074]

[0075] Wherein, x ij , y ij is the x coordinate and y coordinate of the j-th key point of the i-th frame skeleton in the skeleton sequence obtained in S2;

[0076] S32, specifically:

[0077] The x variance of the i-th frame skeleton, denoted as The y variance of the i-th frame skeleton, denoted as

[0078] The and are calculated by the following formula (2):

[0079]

[0080] S33, the average variance of the skeleton sequence, denoted as S 2 , is calculated by the following formula (3):

[0081]

[0082] S34, the x mean of the skeleton sequence, denoted as μ x ; the y mean of the skeleton sequence, denoted as μ y ; the μ x and μ y are calculated by the following formula (4):

[0083]

[0084] S35, the normalized skeleton sequence is calculated by formula (5):

[0085]

[0086] wherein x' ij ,y' ij are the x and y coordinates of the jth key point of the i-th frame of the normalized skeleton sequence.

[0087] The number of the plurality of structure identical spatio-temporal attention encoders is K and the value range of K is 2 to 8.

[0088] The output of the spatio-temporal attention encoder is obtained by adding the output of the temporal attention encoder and the output of the spatial attention encoder.

[0089] The spatial attention encoder is connected in parallel with a self-attention layer and a residual connection, and then connected in series with a layer normalization and a ReLU activation function.

[0090] The temporal attention encoder is composed of a spatial attention encoder, an additional transpose layer before the spatial attention encoder, and an additional transpose layer after the spatial attention encoder.

[0091] The fully connected network is composed of a fully connected layer and a Softmax layer connected in series.

[0092] This embodiment illustrates the process of model training and testing based on the NTU RGB+D 120 dataset using the method of the present application. The NTU RGB+D 120 dataset contains 114480 samples of 120 actions such as falling, sitting, and drinking, each sample containing 4 data sequences of RGB image sequence, depth image sequence, infrared image sequence, and 3D skeleton sequence.

[0093] Figure 2 is the flowchart of model training when the present application is performed, and it can be seen from Figure 2 that the model training process of the image-based lightweight fall detection model has 5 steps, specifically:

[0094] S1, pre-processing the 3D skeleton dataset of NTU RGB+D 120 to obtain a 2D skeleton dataset;

[0095] The pre-processing includes the following sub-steps:

[0096] S1.1, format conversion, removing part of the key points in the 3D skeleton sequence, retaining the head, neck, left and right shoulder joints, left and right elbow joints, left and right wrist joints, left and right hip joints, left and right knee joints, and left and right ankle joints, a total of 14 key points, and re-labeling each key point to obtain a main key point 3D skeleton sequence;

[0097] S1.2, frame the main key point 3D skeleton sequence at intervals of 2 to obtain a 3D skeleton sequence after frame extraction;

[0098] S1.3, project the 3D skeleton sequence after frame extraction to two dimensions through an imaging model; in order to better distinguish falling down and sitting down and crouching down similar to falling down, the samples of falling down, sitting down and crouching down are projected from-15°, 0° and 15° three observation directions, while the samples of other actions are only projected from 0° direction;

[0099] S2, randomly divide the 2D skeleton dataset into a training set, a validation set and a test set according to a ratio of 6:2:2; wherein the training set contains 50245 normal samples and 1616 falling down samples, the validation set contains 16738 normal samples and 549 falling down samples, and the test set contains 16767 normal samples and 520 falling down samples;

[0100] S3, data augmentation processing is performed on the training set to obtain an enhanced training set;

[0101] The data augmentation processing includes the following sub-steps:

[0102] S3.1, random cropping, for samples in the training set with a length greater than 32 frames, a random starting point is specified and cropped to 32 frames, and for samples with a length less than 32 frames, 0 is supplemented to expand to 32 frames;

[0103] S3.2, random flipping, flipping the x-axis of each sample in the training set with a probability of 0.5;

[0104] S4, normalizing the enhanced training set in S3 and the validation set and the test set in S2 to obtain a normalized training set, a normalized validation set and a normalized test set;

[0105] S5, training the spatio-temporal attention network and the fully connected network in the image-based lightweight falling down detection model using the normalized training set in S4 to obtain the trained spatio-temporal attention network and the fully connected network; the loss function adopts the Focal Loss function during training, γ is set to 2 and α is set to 0.8; the optimizer adopts the Adam optimizer, the learning rate is set to 0.002, and the iteration number is set to 15; Figure 3is the loss function value in the model training process of the present application. Among them, the dark gray line represents the loss function value of each Batch of the normalized training set, called training loss. The light gray line represents the loss function value after each iteration (Epoch) of the normalized validation set, called validation loss. It can be seen that the training loss and the validation loss both show a downward trend first fast and then slow. Because the batch sample size (Batch Size) is small, the data of each Batch has a large randomness, and the training loss has a certain large jitter. The number of samples in the validation set is larger, and the randomness is lower, so the validation loss jitter is smaller. The validation loss curve is basically in the average position of the training loss curve, indicating that the model converges well and has no overfitting phenomenon;

[0106] S5, using the normalized test set to test the image-based lightweight fall detection model; Figure 4 is the confusion matrix diagram in the test set of the present application. As can be seen from the figure, the sensitivity of the present application reaches 97.6%, the specificity reaches 97.2%, and the balanced accuracy reaches 97.4%, with a high accuracy;

[0107] Example 2

[0108] This embodiment illustrates that after the image-based lightweight fall detection model of the present application is trained according to example 1, it is deployed to the embedded computer of the mobile robot, and the process of actual application is carried out;

[0109] Figure 5 is the flow chart when the present application is deployed and actually applied. As can be seen from Figure 5 , the model deployment and actual application process of the image-based lightweight fall detection model has 7 steps, which are specifically:

[0110] S1, deploy the fall detection model to the embedded computer;

[0111] The model deployment includes the following sub-steps:

[0112] S1.1, convert the trained Lightweight OpenPose pose estimation model, the trained spatio-temporal attention network and the fully connected network in example 1 into Open Neural Network Exchange (ONNX) format, and copy them to the embedded computer built-in the mobile robot;

[0113] S1.2, import the Lightweight OpenPose pose estimation model, the spatio-temporal attention network and the fully connected network in ONNX format through OpenCV.DNN, and open the OpenCL function to speed up the network inference;

[0114] S2, collecting the image sequence of the old man by the camera built in the mobile robot and storing into the computer;

[0115] S3, obtaining the skeleton sequence of the old man from the image sequence collected in S2 by the Lightweight OpenPose posture estimation model;

[0116] S4, normalizing the skeleton sequence in S3 to obtain the normalized skeleton sequence;

[0117] S5, extracting the deep feature from the normalized skeleton sequence obtained in S4 by using the spatio-temporal attention network;

[0118] S6, judging the fall according to the deep feature obtained in S5 by using the full connection network;

[0119] Figure 6 The effect diagram for fall judgment, in Fig. (6a), when the person in the camera does not fall, the model can identify the skeleton, and judge the action of the person as "normal" through the skeleton, while in Fig. (6b), after the person in the video falls, the model judges the action of the person as "fall" through the skeleton;

[0120] S7, if the old man falls in S7, an alarm is sent out;

[0121] The above is only a preferred embodiment of the present application, and the present application should not be limited to the content disclosed in the embodiment and the drawings. Any equivalent or modification made without departing from the disclosed spirit of the present application falls within the protection scope of the present application.

Claims

1. A lightweight image-based fall detection method, characterized in that, Includes the following steps: S1. Use a camera to capture and store image sequences; S2. Obtain the skeleton sequence from the image sequence acquired in S1 using a pose estimation model; S3. Calculate the mean and variance of the skeleton for each frame, calculate the average variance and mean of the skeleton sequence, and then normalize the skeleton sequence in S2 to obtain the normalized skeleton sequence. S4. Use a spatiotemporal attention network to extract deep features from the normalized skeleton sequence obtained in S3. The spatiotemporal attention network is composed of several spatiotemporal attention encoders with identical structures connected in series. The spatiotemporal attention encoder is composed of a temporal attention encoder and a spatial attention encoder connected in parallel. The spatial attention encoder consists of a self-attention layer and a residual connected in parallel, then sequentially connected in series with a normalization layer and a ReLU activation function. The temporal attention encoder is composed of a transposed layer added before and after the spatial attention encoder; S5. Use a fully connected network to determine the fall based on the deep features obtained in S4.

2. The lightweight fall detection method according to claim 1, characterized in that, The pose estimation model described in S2 is the Lightweight OpenPose pose estimation model, and the skeleton sequence includes N frames of skeleton and 14 key points.

3. The lightweight fall detection method according to claim 2, characterized in that, The range of N is 20 to 60; the 14 key points include the head, neck, left and right shoulder joints, left and right elbow joints, left and right wrist joints, left and right hip joints, left and right knee joints, and left and right ankle joints; each key point includes x-coordinate and y-coordinate.

4. The lightweight fall detection method according to claim 3, characterized in that, S3 includes the following sub-steps; S31. Calculate the mean x and mean y values ​​of the skeleton in each frame of the skeleton sequence; S32. Calculate the x-variance and y-variance of the skeleton for each frame; S33. Calculate the mean variance of the skeleton sequence; S34. Calculate the x-mean and y-mean of the skeleton sequence; S35. Normalize the skeleton sequence in S2 to obtain the normalized skeleton sequence.

5. The lightweight fall detection method according to claim 4, characterized in that, S31, specifically: the mean x-value of the skeleton in the i-th frame, denoted as μ. xi The mean y-value of the skeleton in the i-th frame is denoted as μ. yi ; The μ xi and μ yi Calculated using the following formula (1): Where, x ij ,y ij Let x and y coordinates be the j-th keypoint of the skeleton in the i-th frame of the skeleton sequence obtained by S2; S32, specifically: The variance of x in the skeleton of the i-th frame is denoted as . The variance of y in the skeleton of the i-th frame is denoted as... The and Calculated using the following formula (2): The mean variance of the skeleton sequence described in S33 is denoted as S. 2 It is calculated using the following formula (3): The mean x-value of the skeleton sequence described in S34 is denoted as μ. x The mean y-value of the skeleton sequence is denoted as μ. y The μ x and μ y Calculated using the following formula (4): The normalized skeleton sequence described in S35 is calculated using formula (5): Where, x′ ij ,y′ ij Let x and y be the x and y coordinates of the j-th keypoint in the i-th frame skeleton of the normalized skeleton sequence.

6. The lightweight fall detection method according to claim 5, characterized in that, The number of the several spatiotemporal attention encoders with the same structure described in S4 is K, and the value of K ranges from 2 to 8.

7. The lightweight fall detection method according to claim 1, characterized in that, The outputs of the temporal attention encoder and the spatial attention encoder described in S4 are added together to obtain the output of the spatiotemporal attention encoder.

8. The lightweight fall detection method according to claim 1, characterized in that, The fully connected network described in S5 consists of a fully connected layer and a Softmax layer connected in series.

Images