Human body tumble detection method based on OpenPose

Abstract

The invention belongs to the technical field of picture information processing, provides a human body tumble detection method based on OpenPose, and solves the complex problem in tumble detection. The method comprises the following steps: firstly, inputting a picture processed by a YOLO network into an OpenPose network to extract key point information of a human body, particularly joint positions and trunk characteristics; on the premise that picture processing complexity is not increased, key points and trunk information are directly marked in a picture, so that enhanced extraction of features is realized. According to the method, while the feature expression is enhanced, the recognition accuracy of the detection model on the falling behavior is improved, and the calculation burden of the model is reduced. Through an experiment on a common data set, a result shows that the algorithm can accurately identify a falling behavior in a picture and shows good robustness and practicability. Therefore, the method has a wide application prospect in the field of fall detection, is especially suitable for smart home systems, old-age nursing and other scenes, and has an important practical application value.

Description

Technical Field

[0001] The present invention relates to the technical field of image information processing, and in particular to a human fall detection method background technology based on OpenPose. Background Art

[0002] Falling is one of the most common accidents among the elderly, which may cause serious injuries or even threaten their lives. Therefore, developing efficient fall detection technology has important social significance and economic value.

[0003] In recent years, the research on fall detection can be mainly divided into three categories: wearable device-based, environmental sensor-based, and computer vision-based. Wearable device-based detection technology uses sensors such as accelerometers and gyroscopes to monitor the user's activity status and can send alarms in real time to improve safety. However, wearing devices may affect the user's comfort and acceptance. The environmental sensor-based method relies on devices installed in the environment (such as infrared sensors or pressure sensors) for motion monitoring. It is suitable for long-term monitoring and does not require users to wear devices, but may be affected by occlusion and external interference. Computer vision-based technology analyzes video streams through cameras and image processing algorithms to extract human key points and motion features. Although it can capture rich environmental information, its performance is often limited by lighting conditions and background complexity.

[0004] OpenPose is an advanced human posture estimation technology that can capture the key point information of the human body in the picture in real time, including the position and posture of each joint. This technology is implemented through convolutional neural network (CNN), with high accuracy in human body detection and strong robustness to lighting changes and background interference. The 2D key points output by OpenPose can effectively describe the motion state of the human body and provide reliable feature support for fall detection. Combined with OpenPose's human key point detection technology, the fall detection method based on posture change characteristics can significantly improve the accuracy and real-time performance of detection. This method can quickly determine whether a human body has fallen by analyzing the position information, movement speed and angle changes of the key points. At the same time, with the application of machine learning and deep learning algorithms, the fall detection model can be further optimized and the false alarm rate can be reduced.

[0005] There are the following technical problems:

[0006] (1) When a human body is partially occluded, OpenPose may not be able to accurately identify all key points. In this case, the lack of key points will affect the overall understanding of the human body posture, which in turn leads to the loss of key information. This information loss not only affects the accuracy of posture estimation, but also reduces the reliability of fall detection and increases the risk of misjudgment and missed detection.

[0007] (2) Posture changes and clothing differences between different individuals may significantly affect the generalization ability of the model. When these changes are not taken into account during model training, the posture of new individuals may not be correctly identified, resulting in poor fall detection results. This is especially true for the elderly, whose physical characteristics and movement patterns may be significantly different from the training data, increasing the risk of false positives and false negatives, and reducing the reliability and accuracy of the system in practical applications.

[0008] (3) When using human key point data as features for fall detection, illumination changes and background interference in the original image data may lead to problems such as deviation in key point extraction. Although using only key points to replace other human features can effectively reduce the computational burden, the features of this method are relatively single. If there is a large error in the extraction of key points, it will directly affect the results of fall detection and reduce the accuracy of detection.

[0009] In summary, in order to solve the problems in fall detection, the following measures can be taken. First, enhance the data set to cover a variety of human postures, dressing styles, and environmental conditions, and use data enhancement techniques such as rotation, flipping, and brightness adjustment to improve the generalization ability of the model. Secondly, improve the key point detection algorithm and adopt a more advanced deep learning architecture to improve the accuracy of key point extraction. At the same time, combine RGB images with depth images to reduce the impact of lighting and occlusion on detection. These solutions can effectively improve the performance and reliability of fall detection and provide better protection for the safety monitoring of the elderly. Summary of the invention

[0010] 1. Technical issues to be solved

[0011] In order to solve at least one technical problem existing in the above-mentioned background technology, the present invention proposes a human fall detection method based on OpenPose. This method applies an improved OpenPose network to the fall detection task, which can accurately mark the human posture information in the original image, and realize feature enhancement without increasing the complexity of the image. This method comprehensively utilizes the human key point information and the original image information for fall detection. Even if certain errors occur in the key point extraction process, the original image information can still be used as a reference to assist feature extraction. This effective combination method improves the accuracy of the algorithm in the feature extraction stage, thereby improving the overall detection accuracy of the algorithm.

[0012] (II) Technical solution

[0013] In order to achieve the above-mentioned purpose, the present invention specifically adopts the following technical solutions:

[0014] A human fall detection method based on OpenPose includes the following steps:

[0015] Step 1: Get the image; get the image processed by the YOLO network;

[0016] Step 2: Pose estimation: Use the improved OpenPose network to estimate the human pose of the processed image. The specific steps include: replacing VGG-19 with EfficientNet to reduce the amount of calculation and increase the network's reasoning speed, obtaining the 2D position information of the key points of the human body, and marking the human pose.

[0017] Step 3: Feature extraction: Combining the key point information of the human body and the features of the original image, the layered convolution and multi-head self-attention mechanism are used to enhance the features of the image and extract the deep features related to falls; layered convolution is used for more efficient feature extraction, and the multi-head self-attention mechanism improves the model's ability to perceive subtle changes in human posture;

[0018] Step 4: Fall determination: Based on the extracted features, determine whether the human body is in a falling state; accurately determine the fall by combining key point positions, human body height-to-width ratio, and other indicators;

[0019] Step 5: Model training: Input sample images into the improved OpenPose network for training. During the training process, adjust the learning rate, use weight decay and other optimization techniques to ensure that the model can learn effectively while avoiding overfitting.

[0020] Step 6: Model testing: During the model testing phase, an independent test set is used to verify the trained improved OpenPose network. The images in the test dataset are input into the model to evaluate its performance in the fall detection task.

[0021] Furthermore, in step 2, the original version of OpenPose used VGG-19 as its backbone network. Replacing VGG-19 with EfficientNet as the backbone network can significantly improve the efficiency of the model; EfficientNet significantly reduces the amount of convolution calculation by using depthwise separable convolution.

[0022] Furthermore, in step 3, the standard convolution layer is replaced in the joint point detection module and the feature map extraction part by dividing the input channels in the standard convolution into multiple groups and performing convolution operations separately, thereby reducing the computational complexity while maintaining good feature extraction capabilities.

[0023] Furthermore, in step 3, a multi-head self-attention mechanism is added to the PAF module to help the model model the long-distance dependency between joints and improve the accuracy of the association between joints; the self-attention mechanism can overcome the local problem of convolution and enhance the model's perception of long-distance associations between human body parts through global information interaction;

[0024] Specifically, the self-attention mechanism can capture the interactions between joints across long distances, such as the connection between hands and shoulders, and feet and knees. The multi-head self-attention mechanism is refined on this basis, using multiple independent attention heads to learn different types of relationships between joints from different subspaces.

[0025] Furthermore, in step 4, the process of human posture estimation involves generating confidence maps and part affinity domains of body parts to represent the correlation between the parts. At each stage of processing, the model generates a 2D vector for each limb and each position in the image to encode the direction and position of the limb. The confidence map reflects the probability of the existence of the joint point, and the PAF describes the connection relationship between different joint points. In the reasoning stage, the model comprehensively considers the confidence map and PAF information, and finally outputs the 2D key points of all human parts in the image, thereby realizing the estimation of human posture.

[0026] Furthermore, by binarizing the RGB image in step 4, OpenPose can effectively reduce the computational complexity in the feature extraction stage, enhance the contrast between the joint points and the background, and reduce noise interference, ultimately improving the model's reasoning efficiency and the accuracy of pose estimation.

[0027] (III) Beneficial effects

[0028] Compared with the existing technology, a human fall detection method based on OpenPose has the following beneficial effects:

[0029] This method replaces VGG-19 in OpenPose with EfficientNet to integrate it into the fall detection task. OpenPose can annotate the human posture information in the original image and perform feature enhancement without increasing the complexity of the image, thereby improving the effectiveness of the algorithm in feature extraction. The algorithm can rely on both human key point information and original image information to detect falls. Even if certain errors occur in the key point extraction process, the original image information can still be used as a reference to enhance the stability of feature extraction. In addition, this method effectively avoids the detection result errors caused by key point extraction deviations and unclear features when only using the original color image, thereby improving the detection accuracy of the classification algorithm, and ultimately enhancing the overall detection performance of the classification network, making the detection results more accurate.

[0030] The present invention adds a multi-head self-attention mechanism to the PAF module and processes the input information in parallel, so as to extract features from multiple angles and improve the comprehensiveness of the model. Secondly, it enhances the model's attention to important information, thereby improving the accuracy of classification, recognition or generation tasks. In addition, it can effectively model the dependencies between distant elements in the input sequence, making it more robust when processing long sequences or complex structures.

[0031] The present invention replaces the standard convolution with layered convolution in the PAF module. First, by decomposing the standard convolution operation into multiple simpler steps, such as channel-by-channel convolution and point-by-point convolution, the amount of calculation is greatly reduced and the complexity of the model is reduced. Secondly, it can effectively maintain or even enhance the feature extraction capability. While reducing parameters and computational costs, it can still retain sufficient spatial information and feature representation capabilities to ensure that the accuracy of joint point extraction will not be affected. This enables OpenPose to achieve a more lightweight deployment while maintaining high detection accuracy and adapt to various complex pose estimation tasks. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] Figure 1 It is the overall algorithm framework diagram of the present invention;

[0033] Figure 2 It is the overall algorithm flow chart of the present invention;

[0034] Figure 3 This is the improved OpenPose framework diagram of the present invention;

[0035] Figure 4 It is a schematic diagram of the extraction position of key points of the human body of the present invention;

[0036] Figure 5 It is the detection diagram output by the present invention. DETAILED DESCRIPTION

[0037] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0038] The overall idea of ​​the present invention is: a human fall detection method based on OpenPose, with an improved network framework. First, the preprocessed image is input into the improved OpenPose network to extract the key point features of the human body, predict the confidence map and part affinity domain of the limb parts, and then generate a complete human posture by associating these parts, and output a picture annotated with key points and torso information. Without increasing the complexity of the image, feature enhancement is completed, and the efficiency of feature extraction is improved. Feature extraction is further optimized by introducing deep separable convolution, and finally the improved model is used for actual detection tasks. The detection accuracy of this method on the same data set is better than that of existing algorithms. OpenPose generates data annotated with key points of the human body, and combines key point information and original image data to achieve feature enhancement effects, thereby effectively improving detection performance.

[0039] Embodiment 1

[0040] like Figure 1-5 As shown, a human fall detection method based on OpenPose proposed in one embodiment of the present invention includes the following steps:

[0041] like Figure 3As shown in the figure, a feature extraction network is constructed; the core idea of ​​target recognition is to automatically identify and classify specific objects in pictures or videos through computer vision technology. The core idea of ​​target recognition is to accurately identify and classify targets in pictures through feature extraction, classification and learning, combined with contextual information and real-time processing capabilities; first, data collection is the first step in target recognition, which usually involves obtaining a large amount of labeled image data from public data sets or custom annotations. The data preparation stage not only includes data cleaning and screening, but also data enhancement to improve the generalization and robustness of the model. The second is feature extraction. Traditional methods rely on manually designed features, while modern deep learning methods generally use convolutional neural networks. CNN automatically learns image features through multi-layer convolution operations, thereby capturing important information such as the shape, color and texture of the object. Convolution kernels at different layers can extract features at different levels, giving the model stronger expression capabilities. After feature extraction is completed, the target recognition model needs to be trained. Usually, supervised learning methods are used to optimize model parameters using labeled data. During the training process, a loss function is used to measure the difference between the predicted results and the true labels, and the model parameters are updated through optimization algorithms. In order to improve the accuracy and stability of the model, commonly used methods such as data enhancement, regularization technology and cross-validation are widely used. After the model is trained, the extracted features are mapped to specific categories through the classifier. Commonly used classifiers include linear classifiers and fully connected layers in deep learning. The post-processing stage involves threshold setting and non-maximum suppression (NMS) to optimize the detection results, eliminate redundant detection boxes, and improve the final recognition accuracy. This process can effectively reduce the false alarm rate and improve the practicality of the model. Performance evaluation is an indispensable part of the target recognition system. Commonly used evaluation indicators include accuracy, recall rate, F1 score and mean average precision (mAP). These indicators not only reflect the actual performance of the model, but also provide a basis for subsequent model improvements.

[0042] EfficientNet optimizes the network structure through compound scaling, making it perform well on multiple visual tasks and significantly improving the accuracy of the model. Compound scaling can uniformly adjust the depth, width and input resolution of the network to achieve the best results. The public announcement of EfficientNet is as follows:

[0043] Scale(d,w,r)=α d ×β w ×γ r (1)

[0044] Among them, d represents the depth of the network, w represents the width of the network, r represents the resolution of the input image, and α, β, and γ are the corresponding scaling factors.

[0045] EfficientNet has fewer parameters than VGG-19, which gives it a significant advantage in memory usage and computational overhead. For example, the number of parameters of EfficientNet-B0 is only about 5.3M, while VGG-19 has about 143.67M parameters. The reduction in the number of parameters means that fewer computing resources can be used during training and inference. EfficientNet introduces depthwise separable convolution, which further improves the computational efficiency of convolution operations. The formula is:

[0046]

[0047] Where K i is the convolution kernel of each channel, x i It is the i-th channel of the input feature map. This structure splits the standard convolution into depth convolution and point-by-point convolution, reducing the computational complexity.

[0048] The design of EfficientNet allows the scale and complexity of the network to be flexibly adjusted according to specific task requirements, making it adaptable to different application scenarios. This flexibility is reflected in the ability to select EfficientNet models of different sizes based on resource constraints to achieve optimal performance on different platforms.

[0049] Embodiment 2

[0050] Part Affinity Fields (PAF) is one of the key modules of OpenPose for human pose estimation, which is mainly used to describe the association between human joints. In PAF, the 2D vector of each pixel position represents the possibility that the position belongs to a human joint, thereby capturing the connection information between body joints. However, PAF mainly relies on local convolution operations and has difficulty capturing long-distance global information. Therefore, when processing complex human poses, there may be problems in accurately judging the association between remote joints.

[0051] In the traditional PAF module, the input is usually the feature map output by the convolutional network to generate the affinity field between joints. When the multi-head self-attention mechanism is introduced, these feature maps are regarded as the input of the self-attention mechanism. Assume that the input feature map is Where H and W represent the height and width of the feature map respectively, and C represents the number of channels.

[0052] The self-attention mechanism captures global information by calculating the correlation between different positions in the feature map. The core steps of self-attention include calculating the query (Query, Q), key (Key, K) and value (Value, V), Q = XW Q , K = XW K , V = XW V, where are the linear transformation matrices used to map input features to query, key, and value spaces, respectively.

[0053] In the multi-head self-attention mechanism, the input features are divided into multiple different attention heads, and each head performs independent self-attention calculations. The formula is:

[0054] MultiHead(Q,K,V)=Concat(head1,head2,...,head h )W O (3)

[0055] The query, key, and value of each attention head are generated through different linear transformation matrices, so that different attention patterns are calculated in parallel in multiple subspaces. This enables the model to focus on the global relationship between features from different perspectives.

[0056] The output of self-attention is calculated by the following formula:

[0057]

[0058] Among them, d k is the dimension of the key vector, and softmax is used to calculate the attention weights to model the relationship between different positions.

[0059] After introducing the multi-head self-attention mechanism into the PAF module, the model can better capture the long-range dependencies between different joints of the human body and improve the accuracy of overall posture estimation. The specific process is as follows:

[0060] (1) Through the multi-head self-attention mechanism, the dependency of each pixel position in the feature map with other positions is calculated to obtain a globally enhanced feature representation.

[0061] (2) The enhanced features not only contain the local information captured by local convolution, but also integrate the correlation between distant joints through the attention mechanism.

[0062] (3) When generating affinity domains, these features enhanced by the self-attention mechanism make the connections between joints more accurate, which can significantly improve the accuracy, especially when dealing with complex postures or long-distance joints.

[0063] Through the multi-head self-attention mechanism, the PAF module can model the correlation of long-distance joints, making the joint connection more accurate. Improve the robustness of pose estimation: Even if the joint spacing is large or there is occlusion, the model can infer the reasonable connection between joints through the self-attention mechanism. Parallel multi-attention heads process different subspaces: The multi-head self-attention mechanism can focus on features at different levels at the same time by processing the attention mechanism of multiple subspaces in parallel, thereby improving the expressiveness of the overall model.

[0064] Embodiment 3

[0065] In the standard convolution operation of the PAF module, each pixel of the input feature map is convolved with a convolution kernel of size K×K. The output feature map size is The calculation formula for the standard convolution operation is as follows:

[0066]

[0067] Among them, C in and C out are the number of channels of the input and output feature maps, K is the size of the convolution kernel, and W is the weight of the convolution kernel.

[0068] The computational cost of standard convolution is:

[0069] FLOPs = H × W × C in ×C out ×K 2 (6)

[0070] This calculation method will bring huge computational burden as the number of input and output channels increases.

[0071] Layered convolution is an efficient convolution method in convolutional neural networks, first introduced by MobileNet. It decomposes the standard convolution operation into two independent steps: deep convolution and point-by-point convolution. Deep convolution is to use a separate K×K convolution kernel for each channel to perform convolution, and no longer sum across channels. Point-by-point convolution is to use a 1×1 convolution kernel to fuse feature maps of different channels to complete information exchange between channels. The computational complexity of these two steps is greatly reduced compared to standard convolution.

[0072] The depth convolution formula and calculation formula are as follows:

[0073]

[0074] FLOPs depthwise =H×W×C in ×K 2 (8)

[0075] The point-by-point convolution formula and calculation formula are as follows:

[0076]

[0077] FLOPs pointwise =H×W×C in ×C out (10)

[0078] The total amount of computation (depthwise convolution + pointwise convolution) is:

[0079] FLOPs total =H×W×C in ×(K 2 +C out ) (11)

[0080] Compared with the standard convolution H×W×C in ×C out ×K 2 , the computational effort of layered convolution is greatly reduced, especially in C in and C out In a very big case.

[0081] The PAF module is used to predict the affinity field between human joints, usually by extracting the feature information of joint connections layer by layer through multiple convolutional layers. Although the standard convolution is effective in capturing local and global information, it is computationally intensive and slow in actual deployment.

[0082] Introducing layered convolution into the PAF module can reduce the computational burden in the feature extraction process while maintaining accuracy. The main improvements are as follows:

[0083] Depthwise convolution: In each layer of convolution operation, each channel is convolved separately to retain independent information between joints.

[0084] Point-wise convolution: The features of each joint are combined through 1×1 convolution to ensure that the interaction information between channels is preserved, thereby generating an affinity field between joints.

[0085] Step 5: Pre-training. During the pre-training process, EfficientNet is used as the network backbone. The batch size is set to 2, and the initial learning rate and the learning rate of the backbone network are both set to 2e-6. To prevent overfitting, the weight decay coefficient is 1e-4. In addition, the size of the input image is adjusted to ensure that the network adapts to inputs of different scales. These settings help optimize model performance and improve training results.

[0086] Step 6: Model testing. In the model testing phase, the Fall Detection Dataset was used. The training effect of the method was evaluated based on common evaluation indicators. Figure 5 The results show that the multi-target tracking algorithm proposed in the present invention performs better than the basic algorithm in the test and achieves better results. This shows that the proposed method can achieve higher performance in terms of accuracy and performance when dealing with fall detection tasks, further verifying its effectiveness.

[0087] Finally, it should be noted that the above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art can still modify the technical solutions described in the aforementioned embodiments or replace some of the technical features therein by equivalents. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A human fall detection method based on OpenPose, characterized in that: The following steps are involved: Step 1: Get the image; get the image processed by the YOLO network; Step 2: Pose estimation: Use the improved OpenPose network to estimate the human pose of the processed image. The specific steps include: replacing VGG-19 with EfficientNet to reduce the amount of calculation and increase the network's reasoning speed, obtaining the 2D position information of the key points of the human body, and marking the human pose. Step 3: Feature extraction: Combining the key point information of the human body and the features of the original image, the layered convolution and multi-head self-attention mechanism are used to enhance the features of the image and extract the deep features related to falls; layered convolution is used for more efficient feature extraction, and the multi-head self-attention mechanism improves the model's ability to perceive subtle changes in human posture; Step 4: Fall determination: Based on the extracted features, determine whether the human body is in a falling state; accurately determine the fall by combining key point positions, human body height-to-width ratio, and other indicators; Step 5: Model training: Input sample images into the improved OpenPose network for training. During the training process, adjust the learning rate, use weight decay and other optimization techniques to ensure that the model can learn effectively while avoiding overfitting. Step 6: Model testing: During the model testing phase, an independent test set is used to verify the trained improved OpenPose network. The images in the test dataset are input into the model to evaluate its performance in the fall detection task.

2. According to claim 1, a human fall detection method based on OpenPose is characterized in that: In step 2, the original version of OpenPose uses VGG-19 as its backbone network. Replacing VGG-19 with EfficientNet as the backbone network can significantly improve the efficiency of the model; EfficientNet significantly reduces the amount of convolution calculation by using depthwise separable convolution.

3. According to claim 1, a human fall detection method based on OpenPose is characterized in that: In step 3, the standard convolution layer is replaced in the joint point detection module and the feature map extraction part by dividing the input channels in the standard convolution into multiple groups and performing convolution operations separately, thereby reducing the computational complexity while maintaining good feature extraction capabilities.

4. According to claim 3, a human fall detection method based on OpenPose is characterized in that: In step 3, a multi-head self-attention mechanism is added to the PAF module to help the model model the long-distance dependency between joint points and improve the accuracy of the association between joint points; the self-attention mechanism can overcome the local problem of convolution and enhance the model's perception of long-distance associations between human body parts through global information interaction; Specifically, the self-attention mechanism can capture the interactions between joints across long distances, such as the connection between hands and shoulders, and feet and knees. The multi-head self-attention mechanism is refined on this basis, using multiple independent attention heads to learn different types of relationships between joints from different subspaces.

5. The method for human fall detection based on OpenPose according to claim 4, characterized in that: In step 4, the process of human posture estimation involves generating confidence maps and part affinity domains of body parts to represent the correlation between the parts. At each stage of processing, the model generates a 2D vector for each limb and each position in the image to encode the direction and position of the limb. The confidence map reflects the probability of the existence of the joint point, and the PAF describes the connection relationship between different joint points. In the inference stage, the model comprehensively considers the confidence map and PAF information, and finally outputs the 2D key points of all human parts in the image, thereby realizing the estimation of human posture.

6. The method for human fall detection based on OpenPose according to claim 5, characterized in that: In step 4, by binarizing the RGB image, OpenPose can effectively reduce the computational complexity in the feature extraction stage, enhance the contrast between the joint points and the background, and reduce noise interference, ultimately improving the model's reasoning efficiency and the accuracy of pose estimation.

Images