Human action recognition method based on spatiotemporal interest points and spatiotemporal graph convolution

Abstract

The present invention relates to a human action recognition method based on spatiotemporal points of interest and spatiotemporal graph convolution. First, human action-related videos are collected, and the videos are processed using OPENPOSE to obtain human skeletal joints. Then, an optical flow method is used to establish a Gaussian pyramid to calculate optical flow vectors, and then Gabor filtering and Gaussian filtering are combined to extract spatiotemporal points of interest. The joints output by OPENPOSE are matched with the spatiotemporal points of interest to form new node features, which include the output key point position, confidence, optical flow vector size, time, and optical flow vector angle. Finally, a key point spatiotemporal graph model is constructed based on the human skeletal structure, and data training and testing are completed. The present invention recognizes human actions based on skeletal recognition combined with spatiotemporal points of interest, and at the same time introduces an attention mechanism to perform detection and judgment in both time and space, thereby improving the accuracy of action recognition.

Description

Technical Field

[0001] The present invention relates to the field of behavior recognition, and belongs to a method for human action recognition based on spatiotemporal interest points and spatiotemporal graph convolution. Background Art

[0002] Human skeleton-based behavior recognition technology is a current research hotspot in the field of computer vision and a hot topic both at home and abroad. It has important theoretical value and broad application prospects.

[0003] Visual analysis of human motion is a new frontier topic that integrates multiple disciplines, including pattern recognition, image processing, computer vision, and artificial intelligence. It can be widely applied in many fields and is playing an increasingly important role in many of them. Research on visual analysis of human motion can be traced back to 1973, with Johansson's human motion perception experiment. This experiment placed lights on the joints of the human body and used the light spot information to identify some simple human actions. This problem did not gradually attract attention until the late 20th century. Many scholars at home and abroad have constructed a large number of human motion recognition datasets. Human motion recognition datasets are divided into 2D and 3D datasets. 2D datasets are generally collected using ordinary cameras, while 3D datasets are collected using depth sensors such as Kinect. 3D datasets contain depth information from images, so they contain much more information.

[0004] Currently, there are many methods for processing human action recognition datasets, such as Bobick's motion energy graph algorithm, which uses the region and location of the target action. Laptev's method extends the Harris corners on a 2D image into 3D space, using these corners to detect the local structure of the action in the spatiotemporal dimension. Luvizon's method extracts local spatial features such as relative position and displacement vectors from a subset of skeletal joints, and then aggregates them into fixed-size descriptors. Wu Zuxuan and Samitha Herath et al.'s method, based on Samitha Herath's analysis of deep structures and quantitative analysis commonly used for action recognition, introduces the basic structures of convolutional neural networks (CNNs) and recurrent neural networks (RNNs). L. Lo Presti and M. Cascia conducted in-depth research on 3D skeleton action recognition, summarizing action representation and classification methods and providing some commonly used datasets. However, they did not include the latest graph convolutional network (GCN) methods. Wang Lei, D.Huynh and Piotr Koniusz proposed an action recognition algorithm based on the Kinect dataset, and compared various aspects of the algorithms using the Kinect dataset. Summary of the Invention

[0005] The purpose of this invention is to provide a human action recognition method based on spatiotemporal interest points and spatiotemporal graph convolution, which provides more accurate results for human action detection by accurately modeling in both time and space dimensions and adding an attention mechanism.

[0006] In order to achieve the above object, the technical solutions specifically adopted by the present invention are as follows:

[0007] A human action recognition method based on spatiotemporal interest points and spatiotemporal graph convolution includes the following steps:

[0008] S1. Preprocessing the acquired raw video data related to human body movements;

[0009] S2, extract human skeleton joint points through OPENPOSE algorithm;

[0010] S3, obtaining the spatial location and time information of the spatiotemporal interest point;

[0011] S4, normalization processing;

[0012] S5, OPENPOSE outputs joint points that match spatiotemporal interest points to form new node features;

[0013] S6. Build the ST-GCN spatiotemporal graph convolution model;

[0014] S7, data training;

[0015] S8. Data testing

[0016] S9. After the data test obtains satisfactory indicators, action recognition can be performed on any video.

[0017] Furthermore, in the step S1, the continuous video clips are decomposed into a sequence of static frame images in BMP format, and then grayscale processing is performed to speed up the processing. At the same time, the data set is annotated with action labels, and finally a data set is obtained. The data set includes a training set, a test set and a validation set, with a ratio of 6:2:2.

[0018] Furthermore, in step S2, a pre-trained COCO model for identifying 18 joints is used, which includes the Caffe model file used by OPENPOSE and the network structure description file; the pre-processed video (BMP format) is used as input, and the human skeleton joints are extracted through the OPENPOSE algorithm to obtain the joint point confidence and joint affinity domain predicted by OPENPOSE, and then the joint points of the same person are spliced ​​into a skeleton using a bipartite graph to obtain the position information and confidence of a series of joint points.

[0019] Furthermore, in step S3, the spatiotemporal interest point detection is performed in combination with the optical flow method. During the detection, a Gaussian pyramid is first established: after the Gaussian pyramid is established for each frame of the video, pyramid tracking is performed, and the optical flow on the topmost image of the pyramid is calculated starting from the top layer. Then, based on the calculation result of the topmost optical flow, the initial value of the optical flow of the second top layer is estimated, and the accurate result is calculated and fed back to the next layer until the optical flow of the original image at the bottom layer is calculated. This operation is performed for each frame of the video at the same time; the size and direction of the optical flow vector of each pixel point are calculated for the video frame containing the optical flow information;

[0020] Then detect the spatiotemporal points of interest: use a Gaussian filter to smooth the video optical flow field for spatial filtering, and use Gabor to further process the Gaussian filtered video in the time domain to enhance the texture and edge information in the image and obtain the spatiotemporal feature intensity R(x,y,t) of the pixel point:

[0021] R(x,y,t)=(P*G filter *h ev ) 2 +(P*G filter *h od )2 (5)

[0022] Among them, x and y represent the position of the pixel, h ev and h od is the integral pair of Gabor filter, defined as

[0023]

[0024] Where ω = 4 / τ. The spatiotemporal feature strength R is mainly affected by two parameters σ and τ, which represent the spatial and temporal scales of the interest point detector respectively; G filter is a two-dimensional Gaussian smoothing kernel function;

[0025] The spatiotemporal feature intensity of each pixel is calculated by the above formula, and a threshold is selected so that:

[0026] R(x,y,t)>threshold (9)

[0027] Among them, R(x, y, t) is the spatiotemporal feature intensity of the point; when R is greater than a given threshold, it indicates that there is a spatiotemporal point of interest at that location; the spatial position information and related time information of the point of interest are output, and the size and direction of the optical flow vector of the spatiotemporal point of interest are output at the same time.

[0028] Furthermore, the joint coordinates obtained from the OPENPOSE algorithm are not only related to human body movements, but also to factors such as the position of the person on the screen and the camera shooting angle. To avoid the influence of these factors, in step S4, the optical flow size of the spatiotemporal interest point is normalized and scaled to the range [0, 1]:

[0029]

[0030] Among them, ||Q|| represents the original optical flow size, ||Q|| max and ||Q|| max Represents the maximum and minimum optical flow values ​​of all interest points, and Q is the normalized value.

[0031] Furthermore, in step S5, for the joint points output by OPENPOSE, the corresponding spatial coordinates output in the spatiotemporal interest points are found, and for the matched spatial points, further feature information is obtained, and the matched joint points and spatiotemporal interest point features are spliced ​​together to form a new node feature vector, which includes the output key point position, confidence, optical flow vector size, time, and optical flow vector angle.

[0032] Furthermore, in step S6, the ST-GCN spatiotemporal graph convolution model includes 9 ST-GCN modules, a pooling layer, an FC fully connected layer and a softmax layer. Each ST-GCN module includes a spatial graph convolution layer GCN, a BN layer, a RELU activation function module, a temporal convolution layer TCN and an attention module CBAM.

[0033] Furthermore, in the step S7, the Adam optimizer is used as the optimizer for model training, the data obtained in step S5 is fed into the model, and after passing through the GCN and TCN convolution layers, the updated feature maps after convolution are respectively fed into the channel attention mechanism and the spatial attention mechanism through the CBAM attention module. The feature maps after weight update are multiplied, and then forward propagation is performed. The training parameters and weights are adjusted and optimized according to the loss value by back propagation. The loss gradually decreases with each training iteration until the loss of the training set gradually stabilizes. The training of the model is completed and the weight of the model is obtained.

[0034] Furthermore, in step S8, the trained ST-GCN model is trained on the test set to obtain the test results, and the differences between the results and the real action labels are compared to calculate the model's accuracy, recall rate and other evaluation indicators.

[0035] The present invention has the following characteristics and beneficial effects:

[0036] The OPENPOSE joints are matched with spatiotemporal points of interest, and the deep learning algorithm based on ST-GCN (Spatial Temporal Graph Convolutional Networks for Skeleton-Based Action Recognition) spatiotemporal graph convolution is used to realize human action recognition with high accuracy.

[0037] By accurately modeling in both time and space dimensions and adding an attention mechanism, more accurate results are provided for action detection. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] Other features, objects and advantages of the present invention will become more apparent upon reading the detailed description of non-limiting embodiments with reference to the following drawings:

[0039] Figure 1 This is an overall flow chart of a method for human action recognition based on spatiotemporal interest points and spatiotemporal graph convolution according to an embodiment of the present invention;

[0040] Figure 2 OPENPOSE network diagram in an embodiment of the present invention;

[0041] Figure 3 18 joints of the human body in the embodiment of the present invention;

[0042] Figure 4 : This is a ST-GCN network diagram in an embodiment of the present invention;

[0043] Figure 5 Schematic diagram of TCN convolution in an embodiment of the present invention;

[0044] Figure 6 Schematic diagram of the CBAM in an embodiment of the present invention. DETAILED DESCRIPTION

[0045] The present invention is described in detail below in conjunction with specific embodiments. The following examples will help those skilled in the art to further understand the present invention, but are not intended to limit the present invention in any form. It should be noted that, in the absence of conflict, the embodiments of the present invention and the features in the embodiments can be combined with each other.

[0046] like Figure 1 As shown in FIG, a human action recognition method based on spatiotemporal interest points and spatiotemporal graph convolution includes the following steps:

[0047] S1: Obtain 500 sets of raw video data related to human actions, including waving, sitting, standing, and other related actions. Preprocess the raw video data by decomposing the continuous video clips into a sequence of static frames in BMP format. Grayscale the data is then converted to speed up processing. The data set is annotated with action labels to obtain a dataset consisting of a training set, a test set, and a validation set, with a ratio of 6:2:2.

[0048] S2: The OPENPOSE algorithm is used to extract human skeletal joints and perform pose estimation. Developed by Carnegie Mellon University using convolutional neural networks and supervised learning, OPENPOSE is a deep learning-based human pose estimation library based on the Caffe framework. It accurately detects and estimates human joint information from images or videos and can be directly used for joint extraction. Caffe consists of four components: Blob (data storage), Layer (layer), Net (network), and Solver (solver). Due to the varying motion characteristics of workers, their joints constantly change during movement, both due to body movements and center of gravity. The degree of change varies significantly depending on the movement. Therefore, by calculating the change in position of different joints relative to the body center, skeletal features are extracted, and the joint coordinates (x, y) are then obtained. This paper uses a pre-trained COCO model that recognizes 18 joints. This model includes the Caffe model files used by OPENPOSE and the network structure description file, which defines the model's hierarchy and parameters. Applying the dataset directly to the COCO model eliminates the need for training and provides more stable and rapid convergence.

[0049] like Figure 2 As shown in the figure, the preprocessed dataset (in BMP format) is fed into the first 10 convolutional layers of the Visual Geometry Group (VGG) deep convolutional neural network. Processing the input video frames yields a series of feature maps F1. The shape of these feature maps F1 is defined as a four-dimensional tensor whose dimensions are the number of samples (N), image height (h), image width (w), and number of channels (C). The number of channels is 512, representing the number of features extracted at each spatial location. This four-dimensional tensor is then used as input for subsequent network layers.

[0050] The obtained feature map F1 is input into the parallel dual-channel network, and one channel (PCM) is output through the convolutional neural network to predict the confidence map of human joints S (S = (S 1 , S 2 , S 3 ,…,S j ), j represents the number of joints to be detected plus the background, i.e. j = 19), and the other channel (PAF) outputs the joint connection affinity domain L (L = (L 1 , L 2 , L 3 ,…,L c), c represents the number of joint pairs to be detected, c = 38), so that the output joint point confidence and joint affinity domain are estimated simultaneously. Among them, the input of the first stage is the feature map F1 extracted by VGG19, which is fused by three 3*3 convolution kernels and then processed by two 1*1 convolution kernels to obtain S 1 and L 1 , enter the second stage. From the second stage to the end of the sixth stage, the input of stage t includes three parts: the prediction result S of the previous stage t-1 , L t-1 With the feature map F 1 , the convolution kernel of the fusion feature is changed from three 3*3 convolution kernels to five 7*7 convolution kernels. The network input at each stage is:

[0051]

[0052] Among them, ρ t and They are the network structure of stage t, S t and L t Represents the prediction results of PCM and PAF at stage t. At each stage, a convolutional neural network is used to extract Part Confidence Maps (PCM) and Part Affinity Fields (PAF).

[0053] In the final stage of OPENPOSE, the model outputs two feature maps, joint confidence maps and joint affinity domain vector field maps, in the form of (16, h, w, 19) and (16, h, w, 38), respectively. h and w are the height and width of the input image, respectively. 16 represents the number of batched images, 19 represents the number of joint types in the coco dataset (18, i.e., all joints of a human body) + 1 background category, and 38 represents the partial affinity domain field of the limb composed of these 18 joints (each limb is composed of two joints, i.e., the number of joint pairs). According to the characteristics of the human body structure, such as Figure 3 As shown in the figure, there are 19 types of joint confidence maps output, namely nose, neck, right shoulder, right elbow, right wrist, left shoulder, left elbow, left wrist, right hip, right knee, right ankle, left hip, left knee, left ankle, right eye, left eye, right ear, left ear and background.

[0054] After obtaining the joint confidence and joint affinity domain predicted by OPENPOSE, since there are multiple people in the video frame, the joints of the same person are spliced ​​into a skeleton using a bipartite graph. Taking the right hand joint as an example, the right elbow is connected to the right wrist (3→4). First, the joint point set J3 and J4 of all right elbow sequence 3 and right wrist sequence 4 in the joint confidence graph are obtained. Then, the joint affinity domain is combined to calculate the joint affinity of each pair. The connection confidence E of (j3∈{J3},j4∈{J4}):

[0055]

[0056] p(u)=(1-u)d j3 +ud j4 (4)

[0057] Among them, p(u) is the interpolation of two candidate body parts d by uniformly sampling u j3 and d j4 The position obtained by the position is used to obtain the similarity of the two joint points. represents the unit vector from joint point j3 to j4, L c (p(u)) represents the value of the joint affinity vector at position p(u). Then, the Hungarian algorithm is used to match the connection confidence of all right elbow joint points 3 and all right wrist joint points 4. The steps are as follows:

[0058] (1) sorting all joint points 3 and all related nodes 4 according to the connection confidence to form a matrix table;

[0059] (2) Traverse all connections with joint point 3 as the starting point and joint point 4 as the end point;

[0060] (3) If there is a connection whose starting point and end point are not marked, then the connection is added to the final matching connection. If there is a point marked at the starting point or end point, then the connection is discarded.

[0061] The Hungarian algorithm effectively solves the optimization problem of keypoint matching, ensuring that each keypoint is correctly connected to its adjacent keypoints. Finally, OPENPOSE outputs a series of joint point position information and confidence (x, y, acc) and saves it as a .json file.

[0062] S3: Obtaining spatial location and time information of spatiotemporal points of interest

[0063] The detection of spatiotemporal interest points is essentially to obtain interest points by performing maximum value filtering of intensity function on each pixel or position in the image. The present invention combines the Lucas-Kanade (LK) optical flow method to detect spatiotemporal interest points. The process is as follows: Figure 1 shown.

[0064] Assume that the position of a pixel q on image I is (x, y), and its brightness value at time t is I(x, y, t). After time dt, the brightness of the corresponding point becomes I(x+dx, y+dy, t+dt).

[0065] I(x,y,t)=I(x+dx,y+dy,t+dt) (5)

[0066] Under the premise of small motion, use Taylor's formula to expand:

[0067]

[0068] When dt is small enough, o(dt) is 0:

[0069]

[0070] Use u and v to represent the velocity components of the optical flow at this point in the horizontal and vertical directions:

[0071]

[0072] -I t =l x u+l y v (9)

[0073] -I t =[I x I y ] (10)

[0074] This is the basic optical flow constraint equation. However, this equation cannot solve for the two unknown variables u and v. To solve this problem, we can take the pixels in a window and consider them to have the same motion. We can then write n optical flow constraint equations and use the least squares method to solve for the pixel motion.

[0075]

[0076] Let M be

[0077] Solving this formula, we get (M T M)d=M T b

[0078] so:

[0079]

[0080] When the pixel displacement between two frames is large (fast movement), the algorithm will have large errors. Therefore, by building a Gaussian pyramid to reduce the image size, the optical flow vector is also reduced. Using the pyramid structure, the amount of motion is corrected from top to bottom.

[0081] Build a Gaussian pyramid. The higher the pyramid level, the smaller the image and the lower the resolution. Build a Gaussian pyramid: Use the pre-processed video frame data as input as the first layer of the pyramid, which is the bottom layer. Downsample it layer by layer, and apply Gaussian filtering to each layer image to remove high-frequency information:

[0082] I downsanmpled (x,y)=I(x,y)*G filter (13)

[0084]

[0085] Among them, * represents the convolution operation, G filter is a Gaussian filter; σ is the standard deviation of the Gaussian filter, which determines the smoothness of the filter. A larger standard deviation results in a wider filter range and a more pronounced smoothing effect. (a, b) are the spatial coordinates of the filter, representing the filter's location in image space. I(x, y) represents the brightness value of the input image; and * represents a convolution operation. Downsampling the Gaussian-filtered image reduces the image size by half. This downsampling is achieved by taking the value of every other pixel, thereby reducing the image resolution by half. Repeating the Gaussian filtering and downsampling creates the next layer of the pyramid. This process is repeated three times until the top layer of the pyramid is reached and sampling stops, resulting in a four-layer Gaussian pyramid. This process is repeated for each frame of the video, resulting in a Gaussian pyramid for each frame, with the lowest-resolution image at the top and the original image at the bottom.

[0086] Next, pyramid tracking is performed. When tracking, the optical flow of the topmost image of the pyramid is calculated first, and then the initial value of the optical flow of the second top layer is estimated based on the calculation result of the optical flow of the top layer. The accurate result is calculated and then fed back to the next layer until the optical flow of the original image at the bottom layer is calculated. Specifically:

[0087] For two adjacent images I and J, for the pixel (x0, y0) in I, we need to find the pixel (x0+d x y0+d y ) matches it, that is, the brightness value I(x0,y0) matches J(x0+d x y0+d y ) is closest, and the vector d=(d x ,d y ) is the optical flow of point (x0, y0). The image is displayed as the coordinates of the pixel point and the brightness value of the point. The pyramid structure of the image can be represented as I L and J L , L represents the number of pyramid layers, L∈{0,1,2,3}. At the top layer, the optical flow estimation value of the top image is initialized to 0 (since the scaling of the pyramid reduces the optical flow value, the optical flow estimation value at the highest point can be set to 0).

[0088]

[0089] Starting from the top layer, the residual is obtained by minimizing the sum of the matching errors (loss function) in the neighborhood of each point, and the iterative residual is initialized

[0090]

[0091] in, and is the estimated optical flow in the x and y directions of the Lth layer; w x and w y is a 2D neighborhood range, and the integration window size is (2w x +1)×(2w y +1); k represents the number of iterations, traversing the pixels of the image to calculate I L Gradients in the x and y directions:

[0092]

[0093] Use the above calculated gradient to calculate the spatial matrix G and

[0094]

[0095] δ is the difference in brightness between two frames of the same layer image, and the matrix G is reversible.

[0096] In order to find the optimal solution, that is, to find the derivative of the error function, the least squares method is used to solve it so that the error function is minimized. The optimal solution is:

[0097]

[0098] Among them, the matrix G remains unchanged during the iteration, so it only needs to be calculated once. It needs to be recalculated each time. After k iterations, it converges to η k When it is less than the threshold, the iteration ends, and after obtaining the top-level residual, the top-level optical flow estimation value g is added. L You can get the top optical flow accurate value d L (Optical flow exact value = optical flow estimation value + residual).

[0099] d L =g L +η k (twenty three)

[0100] The optical flow calculation result (displacement) of the L layer is then fed back to the L-1 layer as the initial optical flow estimation value of the layer.

[0101] g L-1 =2(g L +η k )=2dL (twenty four)

[0102] In g L-1 Based on this, iteratively calculate the residual η of this layer k , the result after the kth iteration plus the optical flow estimation value is the exact value of the optical flow of this layer: and so on until the 0th layer image, which is the bottom layer, to obtain the final result of the optical flow.

[0103] d 0 =g 0 +η k (25)

[0104] After iterating to the final pyramid level, this operation is performed on each frame of the video to obtain the video's optical flow vector field, represented as P(h*w*2*T), where h*w represents the image frame size, 2 represents the horizontal u and vertical v components of each pixel, and T represents the number of video frames. The final output video frame contains the motion trajectory calculated by the optical flow algorithm.

[0105] For video frames containing optical flow information, the magnitude ||Q|| and direction θ of the optical flow vector of each pixel are calculated. The larger the optical flow vector, the more intense the pixel movement:

[0106]

[0107] θ=arctan(u,v) (27)

[0108] Then, the spatiotemporal interest points are detected and the video optical flow field is smoothed using a Gaussian filter for spatial filtering. The Gaussian filtered video is further processed using Gabor in the time domain to enhance the texture and edge information in the image and obtain the spatiotemporal feature intensity R(x, y, t) of the pixel point:

[0109] R(x,y,t)=(P*G filter *h ev ) 2 +(P*G filter *h od ) 2 (28)

[0110] Among them, x and y represent the position of the pixel, h ev and h od is the integral pair of Gabor filter, defined as

[0111]

[0112] Where ω = 4 / τ. The spatiotemporal feature strength R is mainly affected by two parameters σ and τ, which represent the spatial and temporal scales of the interest point detector respectively; Gfilter is a two-dimensional Gaussian smoothing kernel function.

[0113] The spatiotemporal feature intensity of each pixel is calculated by the above formula, and a threshold is selected so that:

[0114] R(x,y,t)>threshold (32)

[0115] R(x, y, t) is the spatiotemporal feature strength of the point. When R is greater than a given threshold, it indicates the presence of a spatiotemporal point of interest at that location. The algorithm outputs the spatial location information (usually expressed as coordinates) and the associated time information (x, y, t), as well as the magnitude and direction (Q, θ) of the optical flow vector of the spatiotemporal point of interest. Spatiotemporal points of interest help the model focus on data that undergoes dramatic changes in spatiotemporal dimensions. This data reflects important information about the target's motion, allowing for the acquisition of node data that is more closely aligned with the corresponding action.

[0116] S4: Normalization

[0117] The joint coordinates obtained from the above steps are not only related to the human body's movements, but also to factors such as the person's position on the screen and the camera angle. The joint features used for recognition should only be related to the current action and should be normalized. Normalize the optical flow size of the spatiotemporal interest points and scale them to the range [0, 1]:

[0118]

[0119] Among them, ||Q|| represents the original optical flow size, ||Q|| max and ||Q|| max Represents the maximum and minimum optical flow values ​​of all interest points, and Q is the normalized value.

[0120] S5: OPENPOSE outputs joint points that are matched with spatiotemporal interest points to form new node features.

[0121] For the joint points output by OPENPOSE, we search for their corresponding spatial coordinates in the spatiotemporal interest points. For the matched spatial points, we further extract feature information and concatenate the matched joint point and spatiotemporal interest point features to form a new node feature vector with the structure [x, y, acc, Q, t, θ], where x and y are the position coordinates, acc is the confidence score output by OPENPOSE, Q is the magnitude of the keypoint's optical flow vector, and θ is the angle of the optical flow vector. For unmatched joint points, Q and θ in their feature vectors are set to 0.

[0122] S6: Building the ST-GCN spatiotemporal graph convolution model

[0123] The Chinese University of Hong Kong proposed a human action recognition method based on ST-GCN, which introduced the time dimension and enabled the graph convolutional neural network to extract the characteristic information of the key points of the human skeleton in time and space. The entire network model of spatiotemporal graph convolution includes 9 ST-GCN modules (the first 3 ST-GCNs have 64 output channels, the middle 3 ST-GCNs have 128 output channels, and the last 3 have 256 output channels), pooling layers, FC fully connected layers, and softmax layers. Figure 4 As shown in the figure, the ST-GCN module includes a spatial graph convolution layer (GCN), a batch normalization layer (BN), a RELU activation function module, a temporal convolution layer (TCN), and an attention module (CBAM). The attention model (CBAM) is added to ST-GCN. The CBAM module includes a channel attention mechanism and a spatial attention mechanism. The channel attention mechanism is used to enhance the representation of each channel in the feature map, while the spatial attention mechanism is used to enhance the representation of each spatial position in the feature map.

[0124] The connection relationship of the skeleton joints is expressed by the adjacency matrix A through linear transformation. If there is a connection relationship between the nodes, the value is 1, and if there is no connection relationship between the nodes, the value of the corresponding position in the matrix is ​​0. The skeleton key point data of all frame images are merged to form a skeleton data sequence. The data structure of the skeleton data sequence is f in (N×C×T×K×M), where N is the batch size of 16; C is the number of channels, assuming C=6, representing the x, y, confidence, optical flow size, time, and angle of each joint respectively; T is the number of frames; K is the number of skeleton joints; and M is the number of people appearing in each frame of the video. Construct the D matrix, or degree matrix, which is a diagonal matrix consisting of the degrees of the joint points, where the degree is the number of nodes connected to that joint point. Normalize the adjacency matrix to highlight the contribution of different nodes:

[0125]

[0126] Among them, D is the degree matrix, A is the adjacency matrix, A′ is the normalized adjacency matrix, I is the identity matrix, and A+I takes the node's own feature information into account by adding a unit matrix.

[0127] Next, the 18 joint points are grouped: ST-GCN uses a spatial distance partitioning method. The same partition has the same weight vector as the inner product during convolution calculation, so there are different weight vectors for each partition. Partitioning is based on the node distance in the space-time graph, where r represents the average distance between the node and the centroid of the skeleton graph. All areas with a distance of 1 are divided into root nodes, centripetal nodes, and centrifugal nodes, with the corresponding weights of each type of node being 1, 2, and 3 respectively. The spatial partitioning method is as follows:

[0128]

[0129] According to different partitions, different adjacency matrices are constructed, which are divided into root node adjacency matrix, adjacency matrix close to the center node, and adjacency matrix far from the center node. That is, A'(18*18) is divided into

[0130] (3*18*18), corresponding to 3 groups of different weights, and 3 convolution kernels will be used for convolution later.

[0131] First, perform graph convolution GCN and initialize the network parameter weights. The formula for the convolutional network of the spatial structure division strategy is:

[0132]

[0133] Among them, f in is the five-dimensional input feature, f out is the output feature, D is the degree matrix, Represents the adjacency matrix of the graph, W j Represents the weights of different graph convolution kernels, j is a different convolution kernel (j = 1, 2, 3), Represents the convolution operation.

[0134] The ST-GCN unit uses GCN to learn the local features of adjacent joints in space. Based on this, it is necessary to learn the local features of joint changes in time. TCN is used to learn the local features of joint changes in time. The convolution kernel first completes the convolution of a node on all its frames before moving to the next node, such as Figure 5 As shown, green edges represent temporal edges, blue edges represent spatial edges, and the convolution kernel size is 3*1. After completing the convolution of one node, the convolution moves to the next node to continue the operation. This results in the temporal features of the superimposed skeleton point graph. ST-GCN alternates between GCN and TCN, ultimately outputting a feature map F of a five-dimensional vector (N×C×T×K×M).

[0135] The feature map data F output by the convolutional layer is sent to the channel attention mechanism, such as Figure 6 As shown. By aggregating the information of the feature map through global average pooling and maximum pooling, two different descriptors are generated. and Then the two descriptors are fed forward into the shared network MLP, and the two features are concatenated and passed through an activation function to obtain the weight coefficient M c ,Finally, multiply the weight coefficient by F to obtain the updated new feature F′.

[0136]

[0137] in and represents the average pooling and maximum pooling features, and σ represents the activation function.

[0138] At the same time, the output feature map F' is sent to the spatial attention mechanism for maximum pooling and average pooling operations to obtain the maximum value feature map and average value feature map respectively. Then, it is input into the convolution layer and activation function through the connection operation to obtain the weight coefficient M S , and finally apply it to the feature map to obtain the weighted feature representation.

[0139] M s (F′)=σ(f(AvgPool(F′);MaxPool(F′)]))

[0140]

[0141] Among them, σ represents the activation function, f represents the convolution operation, and Denote the average pooling and maximum pooling features, respectively, and F″ denotes the final output. The addition of the attention module makes the model pay more attention to important features, improving the representation ability and performance of the network. A global average pooling operation is performed on the feature data extracted by the 9 spatiotemporal graph convolutional network modules, and the learned deep features are integrated to obtain the information data features after global average pooling. It is sent to the fully connected layer for classification, and the action classification based on motion information is obtained. Finally, the relevant motion classification is evaluated and scored by the softmax classifier.

[0142]

[0143] Among them, x is an m-dimensional vector, x i is the output value of the i-th label, m is the total number of categories, It is the sum of the exponential functions of the output values ​​of m categories. The Softmax function can be used to convert the output values ​​of multiple categories into a probability distribution in the range [0,1] and 1. Softmax(x i ) represents the probability value of the i-th action category of different action information data features, and finally the classification result is obtained through Softmax output.

[0144] S7: Data Training

[0145] After the model is built, we use the cross entropy loss function as the loss function.

[0146]

[0147] Where N is the number of samples, m is the total number of samples, and y i is the true label of the i-th sample, is the probability that the i-th sample is predicted to be the correct class.

[0148] The Adam optimizer is used as the optimizer for model training. During training, forward propagation is performed on each batch of training samples. This forward propagation calculates the loss between the predicted and actual values ​​of the training samples to obtain the loss value. Backward propagation is used to adjust and optimize the training parameters and weights based on this loss value. The loss gradually decreases with each training iteration. Once the loss of the training set gradually stabilizes, the model training is completed and the model weights are obtained.

[0149] S8: Data Testing

[0150] The trained ST-GCN model is trained on the test set to obtain the test results. The difference between the results and the real action labels is compared and the accuracy of the model is calculated. The accuracy is the proportion of samples correctly predicted by the network to the total number of samples:

[0151]

[0152] Recall is the proportion of all actual positive examples that are correctly predicted as positive examples. Its exact formula is:

[0153]

[0154] TP is a true positive example, meaning it is predicted as a positive example when it is actually a positive example; TN is a true negative example, meaning it is predicted as a negative example when it is actually a negative example; FP is a false positive example, meaning it is predicted as a positive example when it is actually a negative example; and FN is a false negative example, meaning it is predicted as a negative example when it is actually a positive example. The performance of this model in recognizing human actions is evaluated using the obtained precision and recall rates.

[0155] S9: After the data test obtains satisfactory indicators, action recognition can be performed on any video.

[0156] Table 1 Human action recognition accuracy

[0157] walk sit down Standing waving clap boxing walk 116 0 4 0 0 0 sit down 0 114 0 0 0 0 Standing 2 4 115 0 0 0 waving 1 1 0 99 7 21 clap 0 0 0 17 101 12 boxing 0 0 0 3 11 86 Accuracy 97 96 97 83 85 72

[0158] The table shows that the algorithm based on spatiotemporal interest points and spatiotemporal graph convolution has a high accuracy rate for human action recognition. The accuracy rate for actions such as walking, sitting, and standing has reached more than 95%, achieving good results.

[0159] The above shows and describes the basic principles, main features, and advantages of the present invention. Those skilled in the art should understand that the present invention is not limited to the above embodiments. The above embodiments and descriptions are merely preferred examples of the present invention and are not intended to limit the present invention. Various changes and improvements may be made to the present invention without departing from the spirit and scope of the present invention. Such changes and improvements fall within the scope of the present invention. The scope of protection claimed in the present invention is defined by the appended claims and their equivalents.

Claims

1. A human action recognition method based on spatiotemporal interest points and spatiotemporal graph convolution, characterized by: The following steps are involved: S1. Preprocessing the acquired raw video data related to human body movements; S2, extract human skeleton joint points through OPENPOSE algorithm; S3, obtaining the spatial location and time information of the spatiotemporal interest point; In step S3, the spatiotemporal interest points are detected in combination with the optical flow method. During the detection, a Gaussian pyramid is first established: after the Gaussian pyramid is established for each frame of the video, pyramid tracking is performed, and the optical flow on the topmost image of the pyramid is calculated starting from the top layer. Then, the initial value of the optical flow of the second top layer is estimated based on the calculation result of the optical flow of the top layer. The accurate result is calculated and then fed back to the next layer until the optical flow of the original image of the bottom layer is calculated. This operation is performed for each frame of the video at the same time; the size and direction of the optical flow vector of each pixel point are calculated for the video frame containing optical flow information; Then detect the spatiotemporal points of interest: use a Gaussian filter to smooth the video optical flow field for spatial filtering, and use Gabor to further process the Gaussian filtered video in the time domain to enhance the texture and edge information in the image and obtain the spatiotemporal feature intensity R(x,y,t) of the pixel point: R(x,y,t)=(P*G filter *h ev ) 2 +(P*G filter *h od ) 2 (1) Among them, x and y represent the position of the pixel, h ev and h od is the integral pair of Gabor filter, defined as Among them, ω = 4 / τ; the spatiotemporal feature strength R is mainly affected by two parameters σ and τ, which represent the spatial and temporal scales of the interest point detector respectively; G filter is a two-dimensional Gaussian smoothing kernel function; Calculate the spatiotemporal feature intensity of each pixel and select a threshold value such that: R(x,y,t)>threshold (5) Among them, R(x, y, t) is the spatiotemporal feature intensity of the point; when R is greater than a given threshold, it indicates that there is a spatiotemporal interest point at that location; the spatial location information and related time information of the interest point are output, and the size and direction of the optical flow vector of the spatiotemporal interest point are output at the same time; S4, normalization processing; S5, OPENPOSE outputs joint points that match spatiotemporal interest points to form new node features; In step S5, for the joint points output by OPENPOSE, the corresponding spatial coordinates are found in the spatiotemporal interest points. For the matched spatial points, further feature information is obtained, and the matched joint points and spatiotemporal interest points are spliced ​​together to form a new node feature vector. The new node feature vector includes the output key point position, confidence, optical flow vector size, time, and optical flow vector angle; S6. Build the ST-GCN spatiotemporal graph convolution model; S7, data training; S8. Data testing S9. After the data test obtains satisfactory indicators, action recognition can be performed on any video.

2. The human action recognition method based on spatiotemporal interest points and spatiotemporal graph convolution according to claim 1, characterized in that: In step S1, the continuous video clips are decomposed into a sequence of static frames in BMP format, and then grayscale processing is performed to speed up the processing. At the same time, the data set is annotated with action labels to finally obtain a data set, which includes a training set, a test set, and a validation set in a ratio of 6:2:

2.

3. The human action recognition method based on spatiotemporal interest points and spatiotemporal graph convolution according to claim 1, characterized in that: In step S2, first, the preprocessed video is used as input, and the human skeleton joints are extracted using the OPENPOSE algorithm to obtain the joint confidence and joint affinity domain predicted by OPENPOSE. Then, the joints of the same person are spliced ​​into a skeleton using a bipartite graph to obtain the position information and confidence of a series of joints.

4. The method for human action recognition based on spatiotemporal interest points and spatiotemporal graph convolution according to claim 1, wherein: In step S4, the optical flow size of the spatiotemporal interest point is normalized and scaled to the range [0, 1]: Among them, ||Q|| represents the original optical flow size, ||Q|| max and ||Q|| max Represents the maximum and minimum optical flow values ​​of all interest points, and Q is the normalized value.

5. The method for human action recognition based on spatiotemporal interest points and spatiotemporal graph convolution according to claim 1, wherein: In step S6, the ST-GCN spatiotemporal graph convolution model includes 9 ST-GCN modules, a pooling layer, an FC fully connected layer, and a softmax layer. Each ST-GCN module includes a spatial graph convolution layer GCN, a BN layer, a RELU activation function module, a temporal convolution layer TCN, and an attention module CBAM.

6. The method for human action recognition based on spatiotemporal interest points and spatiotemporal graph convolution according to claim 1, wherein: In step S7, the data obtained in step S5 is fed into the model. After passing through the GCN and TCN convolutional layers, the updated feature maps after convolution are respectively fed into the channel attention mechanism and the spatial attention mechanism through the CBAM attention module. The feature maps after weight update are multiplied, and then forward propagation is performed to optimize the training parameters according to the loss value.

7. The method for human action recognition based on spatiotemporal interest points and spatiotemporal graph convolution according to claim 1, wherein: In step S8, the trained ST-GCN model is trained on the test set to obtain the test results. The differences between the results and the real action labels are compared to calculate the accuracy and recall of the model.

Images