Behavior detection method based on lightweight OpenPose spatiotemporal graph network

Abstract

The present invention discloses a behavior detection method based on a lightweight OpenPose spatiotemporal graph network, comprising the following steps: (1) collecting a data set and preprocessing the image; (2) feeding the data set into the lightweight OpenPose network to obtain a human skeleton sequence; (3) feeding the human skeleton sequence into a DST-GCN network to extract spatial structure features and temporal trajectory features from the spatial and temporal dimensions to form high-level spatiotemporal features; (4) using a Softmax classifier to classify the high-level spatiotemporal features into corresponding action categories; and (5) determining the action category of the test image. The present invention first lightweights OpenPose to improve the real-time performance of the model detection, and at the same time improves the ST-GCN by adopting a dense connection mechanism to enhance the ability of the spatiotemporal convolutional layer to extract long-range correlation information and improve the judgment accuracy.

Description

Technical Field

[0001] The present invention relates to the technical field of abnormal behavior detection using computer vision, and in particular to a behavior detection method based on a lightweight OpenPose spatiotemporal graph network. Background Art

[0002] In daily life, there are two main reasons why people fall. One is tripping or slipping due to inconvenient legs and feet, and the other is falling due to illness. If you do not receive timely help when you fall, it will often lead to worsening injuries or even cost your life. Therefore, it is particularly important to detect human falls.

[0003] Currently, common fall detection technologies can be categorized into three types: wearable, environmental sensor-based, and computer vision-based. Wearable approaches place sensors on belts or watches, but elderly people may forget to wear them due to memory loss or discomfort. Environmental sensor-based approaches, such as infrared monitoring, are associated with some elderly people's allergies to infrared radiation, which can negatively impact their health. Computer vision-based fall detection utilizes cameras to collect human video images, processes them using image processing techniques to extract human features, and then analyzes the body's motion state. Falls can generally be identified using a variety of modalities, such as appearance, optical flow, depth, and the human skeleton. The dynamic human skeleton often conveys important information, and the OpenPose model is currently commonly used for skeleton extraction. While this model boasts the best performance across most detection metrics, its application in real-world scenarios still suffers from limitations such as poor real-time detection, excessive model parameters, and excessive model size. Furthermore, skeleton data has a non-Euclidean structure, and previous convolutional and recurrent neural networks ignore crucial inter-node correlations, resulting in limited overall improvement. Summary of the Invention

[0004] Purpose of the invention: This invention improves OpenPose by making it lightweight and optimizes the ST-GCN network by adopting a dense connection mechanism, thereby improving the accuracy and real-time performance of human behavior detection.

[0005] Technical solution: A behavior detection method based on a lightweight OpenPose spatiotemporal graph network, including the following steps:

[0006] (1) Collect data sets and preprocess images;

[0007] (2) Send the dataset into the lightweight OpenPose network to obtain the human skeleton sequence;

[0008] (3) The human skeleton sequence is fed into the DST-GCN network to extract spatial structure features and temporal trajectory features from the spatial and temporal dimensions to form high-level spatiotemporal features;

[0009] (4) Use Softmax classifier to classify high-level spatiotemporal features into corresponding action categories;

[0010] (5) Determine the action category of the test image.

[0011] The step (2) comprises:

[0012] (2.1) Obtaining the features of the data input into the lightweight OpenPose network;

[0013] (2.2) After feature extraction, the model is fed into the prediction layer of the OpenPose model to obtain the heat map of the key points of the human body and the affinity between different key points. After fusion, the human skeleton sequence is obtained. The 7x7 convolution structure in the prediction layer is replaced by a structure consisting of three convolutions in parallel: 1x7 convolution, 7x1 convolution, and 7x7 convolution. The outputs of the three convolutions are fused after the BN operation. At the same time, a 1x1 convolution is used before the parallel convolution layer to compress the number of feature map channels of the input parallel convolution layer.

[0014] The lightweight OpenPose network uses the MobileNet V1 network to replace the VGG 19 network in the OpenPose model, removes the stride of the conv4_2 / dw layer in the MobileNet V1 network, sets the dilation parameter value to 2, and only uses the first to conv5_5 layers of the MobileNet V1 network.

[0015] In step (3), the DST-GCN network adopts a dense connection mechanism, and designs the nine-layer spatiotemporal graph convolution layer into two dense blocks, the first five layers as one dense block, and the last four layers as one dense block. In each dense block, each layer of spatiotemporal graph convolution is connected to all the previous spatiotemporal graph convolutions, and the features are spliced ​​on the channel across the layers. A transition layer is designed between the two dense blocks to control the complexity of the model, and the number of channels is reduced by a 1×1 convolution layer, and an average pooling layer with a stride of 2 is used to halve the height and width of the feature map.

[0016] The Softmax classifier in step (4) uses two fully connected layers. The first fully connected layer reduces the dimension from 256 to 64 and uses dropout to prevent overfitting. The second fully connected layer reduces the dimension to the number of categories and outputs the behavior classification result.

[0017] The binary cross entropy loss function is used, the L2 regularization term is added, and the Adam optimizer is used to train the optimal model. The target loss function including L2 regularization is:

[0018]

[0019] Among them, L is the target loss function, a is the sample subscript, m is the number of samples, is the sample label, the positive class is 1, the negative class is 0, y a is the probability of predicting a positive result, λ‖θ‖ 2 is the L2 regularization term, θ represents the characteristic coefficient, and λ is the user-specified coefficient.

[0020] In the test phase of step (5), a surveillance video is selected. First, the skeleton sequence of the human target in the surveillance video is obtained through lightweight OpenPose, and then sent to DST-GCN to extract the high-level spatiotemporal feature map of the skeleton sequence through graph convolution and time convolution. The spatiotemporal feature map is then sent to the classifier for classification, and the probability of falling and not falling is output. The one with a higher probability is the judgment result.

[0021] A behavior detection device based on a lightweight OpenPose spatiotemporal graph network includes a memory, a processor, and a computer program stored in the memory and executable on the processor. The device is characterized in that when the computer program is loaded into the processor, the behavior detection method based on the lightweight OpenPose spatiotemporal graph network according to any one of the above items is implemented.

[0022] Beneficial effects: (1) Using a binary cross entropy loss function, we added an L2 regularization term to further avoid model overfitting, and used the Adam optimizer to train the optimal model. This model is suitable for solving problems with large-scale data or parameters, with high computational efficiency and low memory requirements. (2) We lightweighted OpenPose and improved the real-time performance of model detection. (3) We improved ST-GCN by using a dense connection mechanism to enhance the ability of the spatiotemporal convolutional layer to extract long-range correlation information and improve judgment accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] Figure 1 Flowchart of the method of the present invention. DETAILED DESCRIPTION

[0024] The technical solution of the present invention will be further described below with reference to the accompanying drawings.

[0025] like Figure 1 As shown, the present invention provides a technical solution:

[0026] The behavior detection method based on the lightweight OpenPose spatiotemporal graph network includes the following steps:

[0027] (1) Collect data sets and preprocess images;

[0028] (2) Send the dataset into the lightweight OpenPose network to obtain the human skeleton sequence;

[0029] (3) The human skeleton sequence is fed into the DST-GCN network to extract spatial structure features and temporal trajectory features from the spatial and temporal dimensions to form high-level spatiotemporal features;

[0030] (4) Use Softmax classifier to classify high-level spatiotemporal features into corresponding action categories;

[0031] (5) Determine the action category of the test image.

[0032] The step (2) comprises:

[0033] (2.1) Obtaining the features of the data input into the lightweight OpenPose network;

[0034] (2.2) After feature extraction, the model is fed into the prediction layer of the OpenPose model to obtain the heat map of the key points of the human body and the affinity between different key points. After fusion, the human skeleton sequence is obtained. The 7x7 convolution structure in the prediction layer is replaced by a structure consisting of three convolutions in parallel: 1x7 convolution, 7x1 convolution, and 7x7 convolution. The outputs of the three convolutions are fused after the BN operation. At the same time, a 1x1 convolution is used before the parallel convolution layer to compress the number of feature map channels of the input parallel convolution layer.

[0035] The lightweight OpenPose network uses the MobileNet V1 network to replace the VGG 19 network in the OpenPose model, removes the stride of the conv4_2 / dw layer in the MobileNet V1 network, sets the dilation parameter value to 2, and only uses the first to conv5_5 layers of the MobileNet V1 network.

[0036] In step (3), the DST-GCN network adopts a dense connection mechanism, and designs the nine-layer spatiotemporal graph convolution layer into two dense blocks, the first five layers as one dense block, and the last four layers as one dense block. In each dense block, each layer of spatiotemporal graph convolution is connected to all the previous spatiotemporal graph convolutions, and the features are spliced ​​on the channel across the layers. A transition layer is designed between the two dense blocks to control the complexity of the model, and the number of channels is reduced by a 1×1 convolution layer, and an average pooling layer with a stride of 2 is used to halve the height and width of the feature map.

[0037] The Softmax classifier in step (4) uses two fully connected layers. The first fully connected layer reduces the dimension from 256 to 64 and uses dropout to prevent overfitting. The second fully connected layer reduces the dimension to the number of categories and outputs the behavior classification result.

[0038] The binary cross entropy loss function is used, the L2 regularization term is added, and the Adam optimizer is used to train the optimal model. The target loss function including L2 regularization is:

[0039]

[0040] Among them, L is the target loss function, a is the sample subscript, m is the number of samples, is the sample label, the positive class is 1, the negative class is 0, y a is the probability of predicting a positive result, λ‖θ‖ 2 is the L2 regularization term, θ represents the characteristic coefficient, and λ is the user-specified coefficient.

[0041] In the test phase of step (5), a surveillance video is selected. First, the skeleton sequence of the human target in the surveillance video is obtained through lightweight OpenPose, and then sent to DST-GCN to extract the high-level spatiotemporal feature map of the skeleton sequence through graph convolution and time convolution. The spatiotemporal feature map is then sent to the classifier for classification, and the probability of falling and not falling is output. The one with a higher probability is the judgment result.

[0042] A behavior detection device based on a lightweight OpenPose spatiotemporal graph network includes a memory, a processor, and a computer program stored in the memory and executable on the processor. The device is characterized in that when the computer program is loaded into the processor, the behavior detection method based on the lightweight OpenPose spatiotemporal graph network according to any one of the above items is implemented.

[0043] While embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions, and variations may be made to these embodiments without departing from the principles and spirit of the invention, and that the scope of the invention is defined by the appended claims and their equivalents.

Claims

1. A behavior detection method based on a lightweight OpenPose spatiotemporal graph network, characterized by: The following steps are involved: (1) Collect data sets and preprocess images; (2) Send the dataset into the lightweight OpenPose network to obtain the human skeleton sequence; (3) The human skeleton sequence is fed into the DST-GCN network to extract spatial structure features and temporal trajectory features from the spatial and temporal dimensions to form high-level spatiotemporal features; (4) Use Softmax classifier to classify high-level spatiotemporal features into corresponding action categories; (5) Determine the action category of the test image; The step (2) comprises: (2.1) Obtaining the features of the data input into the lightweight OpenPose network; (2.2) After feature extraction, the features are fed into the prediction layer of the OpenPose model to obtain the heat map of the key points of the human body and the affinity between different key points. After fusion, the human skeleton sequence is obtained. The 7x7 convolution structure in the prediction layer is replaced with a structure consisting of three convolutions in parallel: 1x7 convolution, 7x1 convolution, and 7x7 convolution. The outputs of these three convolutions are fused after the BN operation. At the same time, a 1x1 convolution is used before the parallel convolution layer to compress the number of feature map channels of the input parallel convolution layer. The lightweight OpenPose network uses the MobileNet V1 network to replace the VGG 19 network in the OpenPose model, removes the stride of the conv4_2 / dw layer in the MobileNet V1 network, sets the dilation parameter value to 2, and only uses the first layer to the conv5_5 layer of the MobileNet V1 network in the lightweight OpenPose network. In step (3), the DST-GCN network adopts a dense connection mechanism, and designs the nine-layer spatiotemporal graph convolution layer into two dense blocks, the first five layers as one dense block, and the last four layers as one dense block. In each dense block, each layer of spatiotemporal graph convolution is connected to all the previous spatiotemporal graph convolutions, and the features are spliced ​​on the channel across the layers. A transition layer is designed between the two dense blocks to control the complexity of the model, and the number of channels is reduced by a 1×1 convolution layer, and an average pooling layer with a stride of 2 is used to halve the height and width of the feature map.

2. The behavior detection method based on lightweight OpenPose spatiotemporal graph network according to claim 1 is characterized in that: The Softmax classifier in step (4) uses two fully connected layers. The first fully connected layer reduces the dimension from 256 to 64 and uses dropout to prevent overfitting. The second fully connected layer reduces the dimension to the number of categories and outputs the behavior classification result.

3. The behavior detection method based on lightweight OpenPose spatiotemporal graph network according to claim 1 is characterized in that: The binary cross entropy loss function is used, the L2 regularization term is added, and the Adam optimizer is used to train the optimal model. The target loss function including L2 regularization is: Among them, L is the target loss function, a is the sample subscript, m is the number of samples, is the sample label, the positive class is 1, the negative class is 0, y a is the probability of predicting a positive result, λ‖θ‖ 2 is the L2 regularization term, θ represents the characteristic coefficient, and λ is the user-specified coefficient.

4. The behavior detection method based on lightweight OpenPose spatiotemporal graph network according to claim 1 is characterized in that In the test phase of step (5), a surveillance video is selected. First, the skeleton sequence of the human target in the surveillance video is obtained through lightweight OpenPose, and then sent to DST-GCN to extract the high-level spatiotemporal feature map of the skeleton sequence through graph convolution and time convolution. The spatiotemporal feature map is then sent to the classifier for classification, and the probability of falling and not falling is output. The one with a higher probability is the judgment result.

5. A behavior detection device based on a lightweight OpenPose spatiotemporal graph network, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the computer program is loaded into a processor, the behavior detection method based on the lightweight OpenPose spatiotemporal graph network is implemented according to any one of claims 1 to 4.

Images