Real-time fall detection method and system based on enhanced context information

Abstract

The invention discloses a real-time fall detection method and system based on enhanced context information. The method comprises the following steps: 1, acquiring fall-down images and video data to make a data set; 2, preprocessing the data set image; 3, building a fall detection network based on enhanced context information, and constructing a real-time fall detection model; step 4, network training: inputting the data set image into the network, performing forward propagation and back propagation algorithms through a multilayer structure of the network, and repeating for multiple times to obtain network weights and parameters showing optimal performance; 5, tumble detection: inputting a to-be-detected image into the real-time tumble detection model for feature extraction, feature fusion and target frame screening, predicting a tumble behavior in the to-be-detected image, and testing the analysis model; and step 6, outputting a final result of the detection model. The problem that falling behaviors cannot be found in time due to the fact that falling detection cannot be carried out in real time can be solved, and the accuracy is high.

Description

Technical Field

[0001] The present invention belongs to the field of computer vision technology, and in particular relates to a real-time fall detection method and system based on enhanced context information. Background Art

[0002] At present, there are four main types of technical research on fall detection: First, fall detection technology based on wearable sensors: This technology uses sensors worn on the body to detect the movement and posture of the human body to determine whether a fall has occurred. However, this technology currently has problems such as wearing comfort, equipment cost and battery life, and cannot be widely used. Second, fall detection technology based on environmental perception: This technology detects fall events by sensing changes in the surrounding environment. For example, by installing pressure sensors or vibration sensors on the floor, the impact of the human body falling to the ground can be sensed, thereby determining the occurrence of a fall event. This technology is prone to missed detection and false detection, which may cause unnecessary trouble to users, and missed detection will lead to the failure to detect fall events in time, thereby increasing the risk of injury. Third, fall detection technology based on wireless networks: This technology uses wireless network technology to detect fall events by monitoring changes in wireless signals between the human body and the surrounding environment, but is easily interfered by factors such as signal attenuation, multipath effects, and other radio equipment. Fourth, fall detection technology based on computer vision: This technology uses computer vision technology to detect fall events by analyzing video images. The traditional method requires classification and detection of the input video frames, and real-time and accuracy cannot be guaranteed.

[0003] In summary, computer vision-based methods have many advantages. They do not require wearable devices and are not restricted by environmental sensors. They can be detected wherever there is a camera. However, traditional anomaly detection is based on manual feature recognition, which is inefficient and cannot achieve real-time detection of abnormal fall behavior. Summary of the invention

[0004] In order to overcome the above-mentioned deficiencies of the prior art, the object of the present invention is to provide a real-time fall detection method and system based on enhanced context information, which can improve the problem that fall detection cannot be performed in real time, resulting in the inability to detect fall behavior in time, and has high accuracy.

[0005] In order to achieve the above object, the technical solution adopted by the present invention is:

[0006] A real-time fall detection method based on enhanced context information comprises the following steps;

[0007] Step 1, obtain fall image and video data to create a data set;

[0008] Step 2, preprocessing the image of the data set;

[0009] Step 3: Build a fall detection network based on enhanced context information, initialize the weights and parameters of the network, and build a real-time fall detection model for detecting the dataset images preprocessed in step 2;

[0010] Step 4, network training: input the dataset image in step 2 into the network in step 3, and perform forward propagation and back propagation algorithms through the multi-layer structure of the network, repeatedly obtaining the network weights and parameters that show the best performance;

[0011] Step 5, fall detection: Using the optimal network weights and parameters of step 4, the image to be detected is input into the real-time fall detection model for feature extraction, feature fusion, and target box screening, and the fall behavior in the image to be detected is predicted and the analysis model is tested;

[0012] Step 6: Output the final result of the detection model.

[0013] Optionally, the step 2 specifically includes: using Labelimg software to mark the target position, size and category of the input data set image for preprocessing.

[0014] Optionally, the step 3 specifically includes: the real-time fall detection model is an improvement of the YOLOv5 model, and the overall framework of the YOLOv5 model consists of three parts, including a Backbone module, a Neck module and a Prediction module;

[0015] The Backbone module extracts features from the input data set and continuously reduces the feature map;

[0016] The Neck module fuses feature maps of different levels to generate a feature map with multi-scale information, into which the CAM context enhancement module is integrated to achieve the purpose of feature enhancement;

[0017] The Prediction module performs multi-scale processing on the feature map, uses the anchor box to generate the candidate box, predicts the position offset of the candidate box through the regression task, and predicts the probability of the behavior category in the candidate box through the classification task.

[0018] The improvements to YOLOv5 are as follows: the CAM context enhancement module is integrated into the Neck part of the original YOLOv5 network architecture. The CAM module inputs the C5 layer feature map from the FPN structure, performs dilated convolution with different dilated convolution rates, obtains contextual background information of different receptive fields, and injects it into the L1 layer of the PAN structure, which is fused with the features generated by FPN to enrich the contextual information of FPN, achieve the purpose of feature enhancement, and make the network's detection performance better.

[0019] C2, C3, C4, and C5 represent feature maps at different levels, and F1, F2, and F3 are obtained after one layer of convolution operation; L1, L2, and L3 represent the corresponding features output after the features generated by FPN are fused with the results of CAM. The input features are operated by dilated convolutions with dilated convolution rates of 1, 3, and 5, respectively, and the convolution kernel size is 3*3. Then, dilated convolutions are performed on C5 with different dilated convolution rates to obtain different receptive field background information and inject it into FPN to solve the problem that feature maps from different levels will cause semantic conflicts and limit feature expression.

[0020] Optionally, the step 4 specifically includes: using an activation function in a forward propagation algorithm to convert the weighted sum of the input signal into a final output result of the network; using a back propagation algorithm to calculate the loss through a mean square error loss function and using a gradient descent optimization algorithm to update the weight;

[0021] The forward propagation step involves passing the input data to the input layer of the network; computing the weighted sum of the input signals layer by layer and converting them into outputs through activation functions;

[0022] The back-propagation algorithm steps include calculating the loss: calculating the difference between the network's predicted output and the actual label through the loss function; back-propagating the error: starting from the output layer, propagating the error backward to the neurons in each layer according to the gradient of the loss function; updating the weights: based on the back-propagated error, using an optimization algorithm to update the weights of each neuron in the network. By continuously iterating this process, the weights of the network are gradually adjusted to reduce the value of the loss function.

[0023] Optionally, the step 5 specifically includes: testing and analyzing model indicators including F1 value (F1-score), PR curve and AP value;

[0024] Using the optimal network weights and parameters obtained in step 4, the image to be detected is input into the real-time fall detection model to perform feature extraction, feature fusion, target frame screening and other steps to predict the fall behavior;

[0025] The accuracy indicators of the analysis model are F1-score, PR curve and AP value; F1 is the harmonic mean of precision and recall, and the F1 formula is as follows:

[0026]

[0027] Among them, P is precision, which refers to the proportion of samples predicted by the classifier as positive examples that are actually positive examples; R is recall, which refers to the proportion of samples that are actually positive examples that are correctly predicted by the classifier as positive examples; TP, FP, TN, and FN respectively represent true positives, that is, positive examples predicted as positive examples, false positives, that is, negative examples predicted as positive examples, true negatives, that is, negative examples predicted as negative examples, and false negatives, that is, positive examples predicted as negative examples;

[0028] The PR curve is a curve with precision and recall at different thresholds as horizontal and vertical coordinates, and the AP value is the area under the PR curve. The PR curve and AP value evaluate the predictive ability of the model, that is, the performance of the model at different thresholds.

[0029] The present invention also discloses a real-time fall detection system based on enhanced context information, the detection system comprises a video acquisition module, an Input module, a Backbone module, a Neck feature fusion module and a Prediction detection module;

[0030] The video acquisition module is used to shoot and collect image data for the user;

[0031] The Input module collects the collected data set and sends it to the network of the real-time fall detection model;

[0032] The Backbone module is used to extract features from the input data set;

[0033] The Neck feature fusion module realizes the fusion of shallow image features and deep semantic features from Backbone;

[0034] The Prediction detection module is responsible for predicting the location and category of the target and outputting the final result.

[0035] Beneficial effects of the present invention:

[0036] The present invention adds a CAM context enhancement module to the Neck part to form a new composite FPN structure. The Neck part of YOLOv5 adopts the FPN+PAN method. The FPN structure conveys strong semantic features from top to bottom, and the PAN structure allows the feature map to convey strong positioning features from bottom to top. The two structures are combined with each other to perform parameter aggregation on different detection layers from different backbone layers. The CAM module links FPN and PAN, and obtains more context information from the feature map obtained by the FPN structure through different expansion convolution rates, and then merges it with the feature map obtained by the PAN structure, so that the detection feature extraction is more obvious and the accuracy of fall detection is improved.

[0037] Experimental results show that the detection effect of the invention is better, with an accuracy rate of 89.1%, which is 8.7% higher than that of YOLOv5, and can better meet the requirements of real-time fall detection. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] Figure 1 This is a schematic diagram of some fall images in the dataset.

[0039] Figure 2 This is a schematic diagram of the contents of a txt file annotated by Labelimg software.

[0040] Figure 3 This is a schematic diagram of the original YOLOv5 network structure.

[0041] Figure 4 This is a schematic diagram of forward propagation.

[0042] Figure 5 This is a schematic diagram of the slicing operation of the Focus structure.

[0043] Figure 6 It is a schematic diagram of the improved Neck structure provided in the embodiment of the present application.

[0044] Figure 7 It is a schematic diagram of the CAM structure.

[0045] Figure 8 This is a schematic diagram of IOU intersection-and-union ratio calculation.

[0046] Fig. 9 This is a comparison chart of the F1 value of the original YOLOv5 and the improved network.

[0047] Fig.10 It is a schematic diagram of the process of the present invention. DETAILED DESCRIPTION

[0048] The present invention is further described in detail below in conjunction with embodiments.

[0049] Example 1

[0050] like Fig.10 As shown, this embodiment discloses a real-time fall detection method based on enhanced context information, which specifically includes the following steps:

[0051] Step 1: First, we need to collect fall images and video data to create a dataset. In this example, we searched for 9,400 fall images from the online gallery and integrated them into a dataset. Figure 1 Shown are some fall images in the dataset.

[0052] Step 2: Preprocess the data set. The details are as follows:

[0053] The image data is preprocessed by classification and annotation. Labelimg software is used to select the falling object in the image and classify and annotate it as "falling". A txt file is obtained. Figure 2 The file contains five items of data, namely, the classification category, the ratio of the center of the marking box to the length (width) of the image, and the ratio of the length (width) of the marking box to the length (width) of the image.

[0054] Step 3: Build a fall detection network with enhanced context information. The details are as follows:

[0055] Combination Figure 3 The network of this embodiment is built based on the YOLOv5 network. The network includes three modules, namely the Backbone module, the Neck module and the Prediction module. The CAM context enhancement module is integrated into the Neck module of the original YOLOv5 network to inject more context information to improve the model's detection ability for falling behavior.

[0056] This embodiment uses the Backbone module in the network to extract features from the current frame image. Figure 3 The Backbone module includes Focus structure, CBL structure, CSP structure and SPP structure. The Focus structure slices the input image and combines Figure 5 , a 4*4*3 image is sliced ​​into a 2*2*12 feature map; the CBL structure halves the width and height of the feature map each time, downsamples the feature map, and extracts the target features at the same time; the CSP structure mainly splits the feature image into two parts, one part is convolved, and the other part is combined with the result of the convolution operation of the previous part; the SPP structure is spatial pyramid pooling, which can convert feature maps with non-fixed scales into a unified scale, that is, fuse multi-scale features.

[0057] This embodiment improves the Neck module in the network to perform feature fusion operations, combining shallow image features with deep semantic features. Figure 6 , the Neck module includes FPN structure and PANet structure. The FPN structure fuses feature maps from different levels from top to bottom through upsampling and coarser-grained feature map fusion, and from bottom to top through convolutional layers; the PANet structure receives feature maps from different backbone layers of FPN and performs parameter aggregation on them using different detection layers. The improved network integrates the CAM context enhancement module to form a new composite FPN structure, thereby obtaining richer context information and suppressing semantic conflicts in the fusion of feature maps from different levels.

[0058] refer to Figure 6The improved Neck module is based on the classic YOLOv5 network framework and integrates the CAM context enhancement module to form a new composite FPN module. It passes down high-level features and supplements low-level semantics, so that high-resolution and strong semantic features can be obtained, which is conducive to the detection of small targets. C2, C3, C4, and C5 represent feature maps of different levels. After a layer of convolution operation, F1, F2, and F3 are obtained; L1, L2, and L3 represent the corresponding features generated by FPN and the fusion of CAM results. The CAM structure combines Figure 7 The input features are operated through dilated convolutions with dilated convolution rates of 1, 3, and 5, and the convolution kernel size is 3*3. Then, dilated convolutions are performed on C5 with different dilated convolution rates to obtain different receptive field background information and inject it into FPN to solve the problem that feature maps from different levels will cause semantic conflicts and limit feature expression.

[0059] In this embodiment, the Prediction module in the network predicts the fall behavior based on the feature graph. Figure 3 ,This module consists of three 1*1 convolutional layers, where the first convolutional layer is used to reduce the number of channels, reduce the amount of calculation and model parameters, and improve the calculation efficiency of the model; the second convolutional layer is used to further extract features, and the third convolutional layer is used to output the confidence and category information of each detection box.

[0060] This embodiment uses the NMS non-maximum suppression algorithm to deal with the phenomenon that many detection boxes in the results output by the Prediction module point to the same target object; the NMS algorithm first sorts the detection boxes according to the confidence level, retains the box with the highest confidence level, and then Figure 8 , calculate the IOU of other boxes with this box, and delete the detection boxes whose IOU exceeds the threshold until non-overlapping detection boxes are retained.

[0061] Step 4: Perform forward propagation and back propagation algorithms through the multi-layer structure of the network, and repeatedly obtain the optimal network weights and parameters. The details are as follows:

[0062] Combination Figure 4 , the forward propagation step involves passing the input data to the input layer of the network; calculating the weighted sum of the input signals layer by layer and converting them into outputs through activation functions.

[0063] The back-propagation algorithm steps include calculating the loss: calculating the difference between the network's predicted output and the actual label through the loss function; back-propagating the error: starting from the output layer, the error is propagated backward to the neurons of each layer according to the gradient of the loss function. This process uses the chain rule to calculate the error contribution of each neuron; updating the weights: according to the back-propagated error, an optimization algorithm is used to update the weights of each neuron in the network. By continuously iterating this process, the weights of the network are gradually adjusted to reduce the value of the loss function.

[0064] Step 5: Predict the falling behavior and analyze the model accuracy. The details are as follows:

[0065] Using the optimal network weights and parameters obtained in step 4, the image to be detected is input into the real-time fall detection model to perform feature extraction, feature fusion, target box screening and other steps to predict the fall behavior.

[0066] The accuracy indicators of the analysis model are F1-score, PR curve and AP value; F1 is the harmonic mean of precision and recall, which comprehensively considers the impact of precision and recall; the higher the F1 value, the better the performance of the model. The F1 formula is as follows:

[0067]

[0068] Among them, P is the precision, which refers to the proportion of samples predicted by the classifier as positive examples that are actually positive examples; R is the recall rate, which refers to the proportion of samples that are actually positive examples that are correctly predicted by the classifier as positive examples; TP, FP, TN, and FN respectively represent true positives, that is, positive examples are predicted as positive examples, false positives, that is, negative examples are predicted as positive examples, true negatives, that is, negative examples are predicted as negative examples, and false negatives, that is, positive examples are predicted as negative examples.

[0069] The PR curve is a curve with precision and recall at different thresholds as horizontal and vertical coordinates, and the AP value is the area under the PR curve. The PR curve and AP value evaluate the predictive ability of the model, that is, the performance of the model at different thresholds.

[0070] The classic YOLOv5 model and the real-time fall detection based on enhanced context information are used to predict the fall behavior of the same input image. The model training parameters are uniformly set to epoch = 100, batch-size = 100, lr0 = 0.0032. The F1 value comparison of the two models is as follows: Fig. 9 As shown, the accuracy AP is shown in Table 1:

[0071] Table 1 Network accuracy comparison

[0072]

[0073] Combining Table 1 and Fig. 9 It can be seen that the performance of the improved YOLOv5-CAM model is better than the classic YOLOv5 model.

[0074] Step 6: Output the prediction result image after the above steps, which includes the behavior classification of whether the target falls and the confidence of the fall prediction.

[0075] Example 2

[0076] This embodiment discloses a real-time fall detection system based on enhanced context information, which specifically includes a video acquisition module, an input module, a backbone module, a Neck feature fusion module, and a Prediction detection module. Among them, the Input module is responsible for collecting the acquired data set and sending it to the network of the real-time fall detection model; the Backbone module is used to extract features from the input data set; the Neck feature fusion module realizes the fusion of shallow image features and deep semantic features from the Backbone; the Prediction detection module is responsible for predicting the location and category of the target and outputting the final result.

[0077] The implementation of the above modules is based on steps 3 to 5 in Example 1, wherein the examples described are the same as the application scenarios.

[0078] The present invention adopts the method and system described in the above embodiment to predict the falling behavior of a video containing the falling behavior. The specific implementation process is as follows:

[0079] The video acquisition module monitors the screen in real time to collect video data, extracts frames from the video and sends them to the detection network through the Input module. After feature extraction by the Backbone module and feature fusion by the Neck module, the Prediction module predicts the target's falling behavior and outputs the final result.

Claims

1. A real-time fall detection method based on enhanced context information, characterized in that: The steps include: Step 1, obtain fall image and video data to create a data set; Step 2, preprocessing the image of the data set; Step 3: Build a fall detection network based on enhanced context information, initialize the weights and parameters of the network, and build a real-time fall detection model for detecting the dataset images preprocessed in step 2. Step 4, network training: input the dataset image in step 2 into the network in step 3, and perform forward propagation and back propagation algorithms through the multi-layer structure of the network, repeatedly obtaining the network weights and parameters that show the best performance; Step 5, fall detection: Using the optimal network weights and parameters of step 4, the image to be detected is input into the real-time fall detection model for feature extraction, feature fusion, and target box screening, and the fall behavior in the image to be detected is predicted and the analysis model is tested; Step 6: Output the final result of the detection model.

2. A real-time fall detection method based on enhanced context information according to claim 1, characterized in that: The step 2 specifically includes: using Labelimg software to label the target position, size and category of the input data set image and perform preprocessing.

3. The real-time fall detection method based on enhanced context information according to claim 1, characterized in that: The step 3 specifically includes: the real-time fall detection model is an improvement of the YOLOv5 model, and the overall framework of the YOLOv5 model consists of three parts, including a Backbone module, a Neck module, and a Prediction module; The Backbone module extracts features from the input data set and continuously reduces the feature map; The Neck module fuses feature maps of different levels to generate a feature map with multi-scale information, into which the CAM context enhancement module is integrated to achieve the purpose of feature enhancement; The Prediction module performs multi-scale processing on the feature map, uses the anchor box to generate the candidate box, predicts the position offset of the candidate box through the regression task, and predicts the probability of the behavior category in the candidate box through the classification task.

4. The real-time fall detection method based on enhanced context information according to claim 3, characterized in that: The improvements to YOLOv5 are as follows: the CAM context enhancement module is integrated into the Neck part of the original YOLOv5 network architecture. The CAM module inputs the C5 layer feature map from the FPN structure, performs dilated convolution with different dilated convolution rates, obtains contextual background information of different receptive fields, and injects it into the L1 layer of the PAN structure to merge with the features generated by FPN to enrich the contextual information of FPN.

5. The real-time fall detection method based on enhanced context information according to claim 4, characterized in that: C2, C3, C4, and C5 represent feature maps at different levels, and F1, F2, and F3 are obtained after one layer of convolution operation; L1, L2, and L3 represent the corresponding features output after the features generated by FPN are fused with the results of CAM. The input features are operated by dilated convolutions with dilated convolution rates of 1, 3, and 5, respectively, and the convolution kernel size is 3*3. Then, dilated convolutions are performed on C5 with different dilated convolution rates to obtain different receptive field background information and inject it into FPN to solve the problem that feature maps from different levels will cause semantic conflicts and limit feature expression.

6. The real-time fall detection method based on enhanced context information according to claim 1, characterized in that: The step 4 specifically includes: using an activation function in the forward propagation algorithm to convert the weighted sum of the input signal into the final output result of the network; the back propagation algorithm calculates the loss through the mean square error loss function and uses the gradient descent optimization algorithm to update the weight; The forward propagation step involves passing the input data to the input layer of the network; computing the weighted sum of the input signals layer by layer and converting them into outputs through activation functions; The back-propagation algorithm steps include calculating the loss: calculating the difference between the network's predicted output and the actual label through the loss function; back-propagating the error: starting from the output layer, propagating the error backward to the neurons in each layer according to the gradient of the loss function; updating the weights: based on the back-propagated error, using an optimization algorithm to update the weights of each neuron in the network. By continuously iterating this process, the weights of the network are gradually adjusted to reduce the value of the loss function.

7. The real-time fall detection method based on enhanced context information according to claim 3, characterized in that: The step 5 specifically includes: testing and analyzing model indicators including F1 value (F1-score), PR curve and AP value; Using the optimal network weights and parameters obtained in step 4, the image to be detected is input into the real-time fall detection model to perform feature extraction, feature fusion, target frame screening and other steps to predict the fall behavior; The accuracy indicators of the analysis model are F1-score, PR curve and AP value; F1 is the harmonic mean of precision and recall, and the F1 formula is as follows: Among them, P is the precision rate, which refers to the proportion of samples predicted by the classifier as positive examples that are actually positive examples; R is the recall rate, which refers to the proportion of samples that are actually positive examples that are correctly predicted by the classifier as positive examples; TP, FP, TN, and FN respectively represent true positive examples, that is, positive examples predicted as positive examples, false positive examples, that is, negative examples predicted as positive examples, true negative examples, that is, negative examples predicted as negative examples, and false negative examples, that is, positive examples predicted as negative examples; The PR curve is a curve with precision and recall at different thresholds as horizontal and vertical axes. The AP value is the area under the PR curve. The PR curve and AP value evaluate the predictive ability of the model, that is, the performance of the model at different thresholds.

8. A real-time fall detection system based on enhanced context information for implementing the method according to any one of claims 1 to 7, characterized in that: The detection system includes video acquisition module, input module, backbone module, neck feature fusion module and prediction detection module; The video acquisition module is used to shoot and collect image data for the user; The Input module collects the collected data set and sends it to the network of the real-time fall detection model; The Backbone module is used to extract features from the input data set; The Neck feature fusion module realizes the fusion of shallow image features and deep semantic features from Backbone; The Prediction detection module is responsible for predicting the location and category of the target and outputting the final result.

Images