A robust human action recognition method

Abstract

This invention provides a robust human action recognition method. By collecting RGB images of various actions performed by people of different body types in different scenarios and performing pose estimation frame by frame, the key point locations of each person are obtained as sample data. The action categories are labeled based on the key point location distribution to obtain a dataset, which is then divided into a training set and a test set in an 8:2 ratio. A data filter based on human topology is constructed to remove estimated individuals in the image with low similarity to a standard human template. A data association model based on the attention mechanism is designed to extract key point data correlation features. A trained attention model and backbone network model are obtained. The test set samples are input into the pre-trained model to output human action prediction results. This method can be applied to human action recognition problems in noisy conditions and has stronger interference resistance and generalization capabilities.

Description

Technical Field

[0001] The present invention relates to the field of behavior perception technology, and in particular to a robust human motion recognition method. Background Art

[0002] Human action recognition is an important research direction in the field of video understanding. Its main goal is to analyze the sequence of key human skeleton points to accurately identify and understand human actions. This method uses deep learning and computer vision techniques to model the spatiotemporal information of skeleton points, thereby achieving the classification of different actions. It plays an important role in video surveillance, action analysis, human-computer interaction, virtual reality, and other aspects. However, when faced with complex indoor and outdoor scenes, such methods still face the following challenges: the high similarity between the background and the target of the action to be identified, the image acquisition device's lighting conditions, partial occlusion and truncation, etc., which may lead to incompleteness and external noise in the estimated skeleton point data. In addition, existing human action models fail to effectively extract the correlation relationship features between key points.

[0003] In recent years, hand-crafted features and traditional machine learning algorithms have performed poorly with complex movements or highly variable environments. However, deep learning algorithms, particularly those based on recurrent neural networks (RNNs), graph neural networks (GCNs), and convolutional neural networks (CNNs), have made significant progress in data representation and feature extraction of human skeletal point sequences, improving the accuracy of identifying human motion patterns based on skeletal point sequences. Consequently, existing skeletal point-based action recognition methods are primarily based on these deep learning methods.

[0004] Bias in the position estimation of human skeleton points is very easy to occur during the pose estimation process. This is because the image acquisition equipment may be affected by the external environment, such as climate and lighting changes, occlusion and truncation conditions, and may also be affected by the body shape and movement diversity of the target itself, resulting in offset and distortion of the pose estimation. The associated motion between multiple nodes is crucial to distinguishing different human motion patterns, but existing models do not effectively extract and utilize the correlation features between the associated nodes.

[0005] In recent years, a series of deep neural network-based methods for skeletal point action recognition have been proposed. For example, Duan et al. proposed a deep learning framework based on a 3D residual network in Computer Vision and Pattern Recognition (CVPR). Compared with GCN-based methods, this method introduces a Gaussian probability heatmap as the basic representation of skeletal point sequences and uses a confidence-based data filtering method to remove biased data, making the proposed model more robust to noise in pose estimation. However, in real-world scenarios, rapid motion of image acquisition devices, high similarity between objects and backgrounds, changing weather conditions, and occlusion and truncation of people can all introduce significant noise into the upstream pose estimation, causing deviations or even errors between skeletal point positions and the true action. Motion representation methods that rely solely on 3D Gaussian probability heatmaps and confidence-based data filtering methods cannot effectively address estimation biases and errors in multiple keypoints, resulting in low accuracy in human action recognition in practical applications and failing to meet practical requirements. A review of related technologies and practical experimental verification revealed that no human action recognition algorithm can guarantee high accuracy by leveraging the correlations between multiple nodes when target keypoint estimates are biased. Summary of the Invention

[0006] In order to solve the problem that existing human action recognition methods are unable to overcome the sharp drop in recognition accuracy caused by large noise of human skeleton points and weak correlation between key points in actual environments, the present invention proposes a new robust human action recognition method. This method compares the similarity between the human topology structure estimated by the upstream posture estimation network and the predefined human topology structure template, screens out data with large deviations between the estimated human individuals and the template, and designs a data association network based on the attention mechanism to overcome the problem of low correlation of key points, thereby improving the accuracy of action recognition in actual application scenarios.

[0007] The present invention provides a robust action recognition method based on skeleton points. The method includes two stages: human body data denoising based on topological similarity and node correlation analysis and feature extraction based on a channel attention network. Specifically, the method includes the following steps:

[0008] S1. Collect RGB images of people of different body shapes performing specified actions (such as standing, sitting, etc.) in different scenarios. Input the existing human posture estimation network mmpose to estimate the posture frame by frame and obtain the position of the human key points. A set of all key points for each person is used as a sample data. The action category is labeled according to the key point position distribution of each sample to obtain a human body action dataset based on skeleton points. A human body topology template is constructed based on the skeleton point position and relative length statistics of each person in the dataset. The dataset is divided into training and test sets according to the ratio of 8:2.

[0009] The position data list P of each human body's K skeleton points (main joints such as head, shoulders, elbows, wrists, hips, knees and ankles) in the image is [(x1, y1, c1), (x2, y2, c2), ..., (x K ,y K ,c K )] is a sample, and each key point is represented as a triple (x k ,y k ,c k ), k∈[1,K]. Where (x k ,y k ) represents the position coordinates of a person’s key point k in the image, c k Represents the estimated confidence of the point given by the upstream pose estimation algorithm mmpose. The following Gaussian probability distribution preprocessing formula is used to construct a multi-channel map M of a specific size as the model input. Each point in M ​​is calculated according to the following formula:

[0010]

[0011] Where W and H are the sizes of the generated Gaussian probability heat map, generally 640*480 is selected, K represents the number of channels of the Gaussian probability heat map, which is consistent with the number of key points, that is, each key point generates a Gaussian probability heat map of one channel, generally using the convention of COCO key point data, K=17; In addition, i, j, k represent the multi-channel map M i,j,k The width value index, height value index, and channel value index are used; σ controls the radius of the key points on each Gaussian probability heat map between 0 and 36 pixels, and σ is generally set to 0.5. After constructing the Gaussian probability heat map M for each person, the action category corresponding to each Gaussian probability heat map is manually labeled, such as 1 for standing, 2 for squatting, and 3 for lying down. This provides sample data and classification labels for supervised learning. Each sample data contains a Gaussian probability heat map and its corresponding category label. After the pose estimation algorithm mmpose obtains multiple sample data for different actions in the video, it divides the training set and validation set into a ratio of 8:2 to ensure data distribution consistency between the training set and validation set.

[0012] The human topology data template is created based on the established human skeleton point action data set. The backbone length of each sample in the data set is represented by a multi-tuple L = (l1,...,l b ,...,l B ), where b represents the backbone index, B represents the number of backbones, and l represents each bone, such as the length from the left shoulder to the left hip. The multi-tuple is normalized using the maximum value of all bones of each person to obtain the relative length l′ of the backbone b relative to the longest backbone. b :

[0013]

[0014] The relative length template of the human backbone is obtained by taking the average of the relative lengths of the corresponding backbones in all samples in the data set. in, Represents a standard template vector of a human skeleton, represents the mean of the relative lengths of all backbones b in the dataset, i.e., all l′ b The mean of .

[0015] S2. Construct a data filter based on the human topology template, take the human skeleton point sample data estimated by the upstream posture estimation network as input, and calculate the estimated topology structure L and the template topology structure The similarity between the topological structure of the estimated individual in the video and the template topological structure is removed, and the individuals whose similarity is less than the empirical value of 0.45 are removed; the relative length of the backbone of the estimated posture in the test video or image is calculated according to the method of calculating the relative length of the backbone in the dataset, and the inverse of the Euclidean distance is used as the similarity standard α:

[0016]

[0017] The upstream pose estimation network outputs a prediction confidence c for each predicted point k k , the key point estimated confidence c k Less than 0.3, and consistent with the standard human topology template Key point individuals with similarity α less than 0.45 are treated as noise estimates and removed from the action data to be detected.

[0018] S3. Design a data association model based on the attention mechanism to extract the correlation between skeleton key points, obtain the correlation features of skeleton key points, and then fully connect and output the probability of each action;

[0019] Convert the training data sample M to be identified into tensor format and then calculate the key point channel M by normalizing and embedding the Gaussian function k With M k’ Correlation between two channel data:

[0020]

[0021] Among them, θ() and () represents two 1*1 convolutions with different parameter weights, and k and k represent the key point channel indices of a person's Gaussian probability heat map. The above formula represents the correlation between two channels of a sample M. According to the above formula, the correlation between each channel and all other channels is calculated separately and then concatenated to obtain a K*K correlation graph G:

[0022] G=[g(M1,M1),...,g(M1,M K ),...g(M K ,M K )]

[0023] The similarity between different key point channels is normalized using the Softmax layer and multiplied by each key point channel to form the attention map M A :

[0024] M A =softmax(G)M

[0025] Where M represents the original Gaussian heat map of size 640*640*3, M K Represents the Kth channel of the Gaussian heat map. The original Gaussian probability heat map and the channel attention heat map are added using residual connections to obtain the feature fusion map M. F :

[0026] M F =M+M A

[0027] The feature map now contains both the original key point location information and the data association information between channels, which can further extract deep features. ResNet34 is used as the backbone network to extract the features of the fusion map and express them as a probability distribution Z through the Softmax layer:

[0028]

[0029] in, Represents the ResNet34 backbone network, and Z represents the predicted probability of each category of action.

[0030] S4, using the sample data and action labels of the training set, supervises the learning to train the data association and feature extraction network parameters built in S3 to obtain the final network model;

[0031] S5. Input the sample data of the test set into the final trained model and output the prediction results of human body movements, including classified movements and prediction confidence.

[0032] The advantage of this method is that it can accurately identify human movements under noisy conditions. After data processing and model optimization, it can effectively improve the accuracy of human movement recognition based on skeleton points in challenging environments, and the recognition accuracy can be ≥80%. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] The drawings in the specification, which constitute a part of this application, are used to provide further understanding of this application. The schematic implementation of this application and its description are used to explain this application and do not constitute an improper limitation on this application.

[0034] Figure 1 is a flow chart of the skeleton point-based action recognition method of the present invention;

[0035] Figure 2 This is a diagram showing the implementation effect of the present invention on a public dataset of action recognition based on skeleton points;

[0036] Figure 3 This is a diagram showing the effect of the present invention being implemented in a practical application scenario of action recognition based on skeleton points. DETAILED DESCRIPTION

[0037] In order to better understand the technical solution of the present invention, the following further describes the embodiments of the present invention in conjunction with the accompanying drawings and specific examples. Note that the various aspects described below in conjunction with the accompanying drawings and specific examples are only exemplary and should not be understood as limiting the scope of protection of the present invention.

[0038] A flow chart of a robust human action recognition method is as follows Figure 1 As shown in Figure 2, the overall process consists of a topology-based data filter, a channel-based attention network, and a residual connection network. Figure 1 The dotted box in the data part shows the topological denoising process, and the remaining solid box shows the sample data construction process. Figure 1 The solid line frame of the network part is the forward reasoning process of the model, which is as follows:

[0039] Step S1: First, use the camera to collect video data of people of different body shapes performing different actions according to the task requirements, extract key frames from the video data in a uniform sampling manner as image samples, and use the existing pose estimation algorithm mmpose on all image samples to estimate the position of the skeleton key points of each person in each image. Convert the absolute position of the key point in the detection image to the relative position (x′ k , y′ k ):

[0040]

[0041] where x k ,y k is the absolute position of the kth key point, W o , H o is the width and height of the original image. Then, according to the Gaussian probability heat map of a specific size (W, H) to be generated, the actual position (x″) of the key point on the generated Gaussian probability heat map is determined. k , y″ k ):

[0042] x″ k =W*y k '

[0043] y″ k =H*y′ k

[0044] Step S2: Construct a standard human topology template based on the statistical data of the joint positions and the length of the lines between the joints of all samples. Use the inverse of the Euclidean distance as the similarity standard α between the measured data and the human template to screen out individuals with too low similarity to the template.

[0045]

[0046] After filtering out the erroneous individuals whose predicted human body and human topology template have similarities lower than 0.45, the value of each pixel is determined on the Gaussian probability heat map M according to the Gaussian probability formula:

[0047]

[0048] Among them, i, j, and k represent the index values ​​of each dimension of the Gaussian probability heat map respectively.

[0049] Step S3: Design a data association model based on the attention mechanism to extract the correlation between the skeleton key points, obtain the skeleton key point correlation features, and then fully connect and output the probability of each action. The training data samples to be identified are converted into tensor format and input into the feature extraction backbone network. The key point channel M is calculated by the normalized embedded Gaussian function. k With M k’ Data correlation between:

[0050]

[0051] Among them, θ and Represents two different 1*1 convolutions, k and k, representing the different key point channel indices of a person's Gaussian probability heat map. The correlation graph G is calculated for each channel and other channels in turn:

[0052] G=[g(M1,M1),...,g(M1,M K ),...g(M K ,M K )]

[0053] The similarity between different key point channels is normalized using the Softmax layer to obtain the weight of each channel and multiply it by each key point channel to form the attention map M A :

[0054] M A =softmax(G)M

[0055] Where M represents the original Gaussian probability heat map. The original Gaussian probability heat map and the channel attention heat map are added using residual connections to obtain the feature fusion map MF :

[0056] M F =M+M A

[0057] The feature map now contains both the original key point location information and the data association information between channels, which can further extract deep features. ResNet34 is used as the backbone network to extract the features of the fusion map and express them as a probability distribution Z through the Softmax layer:

[0058]

[0059] in, Represents the nonlinear mathematical formula of the ResNet34 backbone, and Z represents the predicted probability of each category of action.

[0060] Step S4: Send the sample data and labels into the network, supervise the learning of network parameters, normalize the prediction results to between 0 and 1 through the Softmax layer, output the prediction probability of each category, and compare the loss with the labeled category. After the gradient backpropagation, the network model weights are cyclically optimized.

[0061] Step S5: Use an image acquisition device to capture an RGB action video of the target. mmpose pose estimation is used to obtain the target's skeletal keypoints. The estimated keypoint locations are not fed directly into the neural network model. Instead, they are first filtered through a topological data filter to remove erroneous estimates with a confidence score below 0.3 and a similarity to the standard human topology template below 0.45. These are then fed into the attention neural network and ResNet34 backbone network to obtain an estimated motion pattern for the estimated keypoint topology structure.

[0062] like Figure 2 As shown in the figure, since the skeleton point data of the public dataset is manually processed and basically has no noise, we use the key point data to generate a Gaussian probability heat map during training, and then send it to the network model to predict the action.

[0063] like Figure 3 As shown in the figure, in the actual test environment, there is a lot of position estimation noise. Therefore, after the camera captures the target's action video, the topological data filter is first used to remove low-quality key point data. Then, a Gaussian probability heat map is generated, and the network weights saved in the training phase are used to obtain the predicted probabilities of different actions.

[0064] The advantage of this method is that it can identify the target's motion pattern based on human skeleton points under noisy conditions. It can screen out erroneous estimated postures in actual outdoor scenes, especially under conditions with high similarity between the background and the target, fast relative motion between the camera and the target, and large influence of the climatic environment, with a recognition accuracy rate of >80%.

[0065] In order to simplify the explanation of the present method, the above figures and texts are described as a series of steps, but it should be understood and appreciated that these methods are not limited by the order of operations. Because one or more steps are implemented, some actions may occur in a different order, but those skilled in the art can understand the principles of their actions.

[0066] Although the illustrative specific embodiments of the present invention have been described step by step so that those skilled in the art can understand them, the present invention is not limited to the scope of the specific embodiments. Those skilled in the art can make various changes or modifications within the scope of the claims, as long as the various changes are within the spirit and scope of the present invention as defined and determined by the appended claims.

Claims

1. A robust human action recognition method, characterized in that: The following steps are involved: S1. Collect RGB images of people of different body shapes performing specified actions in different scenarios. After frame-by-frame pose estimation, obtain the key point positions of each person as sample data. Label the action categories based on the key point position distribution to obtain a dataset. Statistically calculate the lengths of each bone in the dataset to construct a human topology template. The dataset is then divided into a training set and a test set. S2. Build a data filter based on human topology template, input sample data, use Euclidean distance to determine the similarity between the estimated topology structure and the template topology structure, and remove predicted individuals in the video whose estimated topology structure and template topology structure are less than 0.45; S3. Design a data association model based on the attention mechanism to extract the correlation coefficient matrix between the key points of the skeleton, obtain the correlation features of the key points of the skeleton, and then fully connect and output the probability of each action; S4: Use the sample data and action labels of the training set to train the data association and feature extraction network parameters built in S3 to obtain the final model; S5. Input the sample data of the test set into the final trained model and output the predicted results of human body movements.

2. A robust human motion recognition method according to claim 1, characterized in that: The overall framework is divided into a human skeleton point data filter and a human skeleton point data association and feature extraction network. The human skeleton point data filter is used to remove the low-similarity distribution between the single-frame image estimation and the human topology template, providing cleaned data for neural network training and testing. The human skeleton point data association and feature extraction network is used to obtain the correlation relationship between each key point, output the attention feature map and classify it.

3. The robust human motion recognition method according to claim 1, wherein: In the dataset preparation and topology denoising stage, the pose estimation algorithm obtains multiple sample data under different action modes in the video, removes the noise data, and divides the data into training and validation sets; The action recognition algorithm based on skeleton points uses the position data list of K human skeleton points in each frame of the video as P = [(x1, y1, c1), (x2, y2, c2), ..., (x K ,y K ,c K )], k∈[1,K] is the original data, and each key point is represented as a triple (x k ,y k ,c k ); Where (x k ,y k ) represents the position coordinates of key point k in the image, c k Indicates the estimated confidence of the point given by the pose estimation algorithm; the backbone length of each sample in the data set is represented by the multi-tuple L = (l1,...,l b ,...,l B ), where b represents the backbone index, B represents the number of backbones, and l represents each bone; Normalize the multi-tuple using the maximum value of all bones of each person to obtain the relative length l' of the b backbone relative to the longest backbone b : The relative length template of the human backbone is obtained by taking the average of the relative lengths of the corresponding backbones in all samples in the data set. in, Represents a standard template vector of a human skeleton, represents the mean of the relative lengths of all backbones b in the dataset, i.e., all l′ b The mean of ; the relative length of the backbone of the estimated posture in the test video or image is calculated according to the method of calculating the relative length of the backbone in the dataset, and the inverse of the Euclidean distance is used as the similarity standard α: The upstream pose estimation network outputs a prediction confidence c for each predicted point k k , the key point estimated confidence c k Less than 0.3, and consistent with the standard human topology template Key point individuals with similarity α less than 0.45 are treated as noise estimates and removed from the action data to be detected; The following Gaussian probability distribution preprocessing formula is used to construct a multi-channel graph M of a specific size as the model input. Each point in M ​​is calculated according to the following formula: Where W and H are the sizes of the generated Gaussian probability heat map, generally 640*480 is selected, K represents the number of channels of the Gaussian probability heat map, which is consistent with the number of key points, that is, each key point generates a Gaussian probability heat map of one channel, generally using the convention of COCO key point data, K=17; In addition, i, j, k represent the multi-channel map M i,j,k The width value index, height value index and channel value index of σ control the radius of the key point on each Gaussian probability heat map between 0-36 pixels, and take σ = 0.

5.

4. The method for robust human motion recognition according to claim 1, wherein: Correlation analysis channel attention model, converting the training data sample M to be identified into tensor format and then calculating the key point channel M by normalizing and embedding the Gaussian function k With M k‘ Correlation between two channel data: Among them, θ() and Represents two 1*1 convolutions with different parameter weights, k and k, representing the different key point channel indices of a person's Gaussian probability heat map; the above formula represents the correlation between two channels of a sample M. According to the above formula, the correlation between each channel and all other channels is calculated separately and then concatenated to obtain a K*K correlation graph G: G=[g(M1,M1),...,g(M1,M K ),...g(M K ,M K )] The similarity between different key point channels is normalized using the Softmax layer and multiplied by each key point channel to form the attention map M A : M A =softmax(G)M Where M represents the original Gaussian heat map of size 640*640*3, M K Represents the Kth channel of the Gaussian heat map; the original Gaussian probability heat map and the channel attention heat map are added using residual connections to obtain the feature fusion map M F : M F =M+M A The feature map at this time contains both the original key point location information and the data association information between each channel, further extracting deep features; ResNet34 is used as the backbone network to extract the features of the fusion map and is expressed as a probability distribution Z through the Softmax layer: in, Represents the ResNet34 backbone network, and Z represents the predicted probability of each category of action. During the test, the key point data of posture estimation in the video frame is first passed through a data filter based on confidence and human topology structure. The human body data with key point confidence lower than 0.3 and bone relative length similarity lower than 0.45 to the standard bone length established according to the database are first removed, and then sent to the above-mentioned neural network for action mode discrimination. The data association and feature extraction network are iteratively trained, and the training is considered complete when the accuracy reaches more than 80%. After that, the video to be recognized is input into the posture estimation algorithm, and the skeleton key points are obtained and sent to the data filter based on the topology structure. After processing, the data association network dynamically assigns weights to each key point, and finally inputs the feature extraction network based on the residual connection network for classification to realize human action recognition.

Images