A skeleton-based behavior recognition method, system, device and storage medium

Abstract

This paper proposes a skeleton-based behavior recognition method, system, device, and storage medium. This method obtains the human skeleton coordinates captured by a body-sensing camera or extracted from a video using a human pose estimation method. These coordinates are then fed into a graph convolutional neural network (GCNN) integrated with a multi-scale motion activation module. The GCNN then extracts spatial and long-term temporal features of the skeleton's joints, while the multi-scale motion activation module extracts multi-scale short-term motion features of the joints, enhancing the model's temporal representation capabilities.

Description

Technical Field

[0001] The present invention belongs to the field of computer vision, and in particular to a skeleton-based behavior recognition method, system, device and storage medium. Background Art

[0002] Behavior recognition is a fundamental problem in computer vision and plays an increasingly important role in many fields, including intelligent monitoring, smart transportation, human-computer interaction, and healthcare. Behavior recognition methods can be divided into video-based and skeleton-based methods based on the data modality used. In recent years, with the upgrade of hardware (such as the Microsoft Kinect camera) and the improvement of the performance of human pose estimation methods, skeleton-based behavior recognition tasks have received widespread attention and have become a research hotspot. Skeleton-based behavior recognition tasks aim to model the motion patterns of human skeleton sequences, extract the spatiotemporal features of skeleton sequences, and thus identify the behaviors represented by the skeleton sequences.

[0003] Compared to video modalities, skeletons are non-Euclidean structured data, consisting only of two- or three-dimensional coordinates of the positions of human joints. This data has lower dimensionality and is more robust to environmental or lighting changes. Short-term motion information of skeleton joints is crucial for identifying skeletal behaviors, revealing the direction and size of the skeleton's movement. However, existing graph convolutional neural network-based methods focus more on modeling the spatial dependencies of skeleton joints and ignore their motion information. Single-scale motion information makes it difficult to simulate skeletal behaviors at different motion rates, limiting the model's generalization capabilities. Extracting multi-scale short-term motion information to improve the accuracy of skeleton-based behavior recognition remains an open question. Summary of the Invention

[0004] The technical problem to be solved by the present invention is the low accuracy of behavior recognition by existing algorithms.

[0005] In order to solve the above problems, the present invention proposes a skeleton-based behavior recognition method, system, device and storage medium.

[0006] A skeleton-based behavior recognition method, comprising:

[0007] S1, obtain the coordinates of the human skeleton joints, and obtain a skeleton sequence based on the coordinates of the human skeleton joints in multiple frames;

[0008] S2, input the skeleton sequence into the graph convolutional neural network, and extract the spatial features of the skeleton sequence through the graph convolutional neural network Where R is the dimension, B is the batch size, C is the channel dimension, T is the length of the skeleton sequence, and N is the number of nodes of the skeleton joint points of a single frame in the skeleton sequence;

[0009] S3, convolution operation is performed on feature X to obtain two features X1 and X2, the dimensions of X1 and X2 are expanded, the expanded X1 and X2 are adjusted to the same dimension, the two adjusted features are fused to obtain the feature weight M, and the activation weight M is obtained by global averaging. p , M p Perform dot multiplication with X to obtain the activated feature X r ;

[0010] S4, X r Input the temporal convolutional neural network and get a layer including feature X, feature X m and the network output of temporal features;

[0011] S5, repeat S2 to S4 until the network output of all layers is obtained as the final feature of the joint point, the final feature is pooled and input into the classifier, the final feature is mapped to the behavior category through the classifier, and the category is used as the behavior recognition result of the skeleton.

[0012] Preferably, in S3, in the process of obtaining features X1 and X2 through convolution operation of feature X, the convolution kernel channels of the convolution operation are controlled to reduce the number of channels of features X1 and X2, Where r is the channel number reduction factor, and r is a positive integer.

[0013] Preferably, in S3, the dimension of X1 is expanded to [B, C, T, 1, N], and the dimension of X2 is expanded to [B, C, 1, T, N]; through the matrix broadcast mechanism, the expanded X1 and X2 are adjusted to the same dimension.

[0014] Preferably, in S3, fusion is performed using the formula M=σ(X1-X2), where σ(·) is a hyperbolic tangent activation function.

[0015] Preferably, in S3, M p Multiply by X to get X m , for X m After the residual connection feature X, the activation feature X is obtained r , that is, X r =X m +X.

[0016] Preferably, in S3, the activation weight M p Then, a 2D convolutional layer is used to transform M p Restore to the same channel dimension as feature X.

[0017] A skeleton-based behavior recognition system is used to implement the above-mentioned skeleton-based behavior recognition method, including an acquisition module for obtaining the human skeleton coordinates captured by a somatosensory device, or using a human posture estimation method to extract the human skeleton coordinates in a video; a spatial relationship extraction module for extracting the spatial features of the skeleton sequence; a motion activation module for extracting activation features and temporal features; and a classification module for outputting the skeleton behavior recognition results.

[0018] A skeleton-based behavior recognition device includes a processor and a memory, wherein the memory is used to store at least one program, and the processor is used to execute the above-mentioned skeleton-based behavior recognition method.

[0019] A skeleton-based behavior recognition storage medium is used to store instructions or programs for implementing the above-mentioned skeleton-based behavior recognition method.

[0020] Compared with the prior art, the present invention has the following advantages and effects:

[0021] This paper uses a graph convolutional neural network to extract spatial and long-term temporal features of skeleton joints, and a multi-scale motion activation module to extract multi-scale short-term motion features of skeleton joints, improving the model's temporal representation capabilities. The target network effectively utilizes the motion information of skeleton joints, possessing stronger temporal representation capabilities, and thus improving the accuracy of skeleton-based action recognition results. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0023] Figure 1 It is a flow chart of the method of the present invention.

[0024] Figure 2 Schematic diagram of the multi-scale motion activation module structure of the present invention.

[0025] Figure 3 This is a schematic diagram of the insertion position of the multi-scale motion activation module of the present invention. DETAILED DESCRIPTION

[0026] In order to make the purpose, technical solutions and advantages of the present invention more clearly understood, the present invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0027] Example 1: A skeleton-based behavior recognition method, comprising:

[0028] Step 1: Obtain the coordinate data of the human skeleton joint points.

[0029] Use a somatosensory camera (such as the Microsoft Kinect camera) to obtain the coordinates of the human skeleton joints, or use an ordinary camera to shoot a video and use existing human pose estimation methods (such as OpenPose) to extract the coordinates of the human skeleton joints in the video. The skeleton sequence is obtained based on the coordinates of the human skeleton joints in multiple frames of images.

[0030] Step 2: Extract the spatial features of skeleton joints.

[0031] To extract the spatial dependencies of skeleton joints, you can choose any existing graph convolutional neural network based on the application scenario. For example, for scenarios requiring high accuracy, you can choose the CTR-GCN network, while for scenarios requiring high real-time performance, you can choose the Shift-GCN network. Using existing algorithms, the coordinates of the skeleton joints are input into the graph convolutional neural network to model the spatial dependencies of the skeleton joints and extract the spatial features of the skeleton sequence.

[0032] Step 3: Extract the short-term motion features and long-term temporal features of the skeleton joints.

[0033] 3.1 Extracting short-term motion features of skeleton joints

[0034] like Figure 2 The multi-scale motion activation module shown can extract the short-term motion features of the skeleton joints. Given the input node features Where B is the batch size, C is the channel dimension, T is the length of the skeleton sequence, and N is the number of skeleton joint nodes in a single frame of the skeleton sequence. Preferably, in order to reduce the number of parameters and computations and simulate the motion of the skeleton joints, two 1×1 2D convolutional layers can be used to reduce the number of feature channels, that is, two 1×1 2D convolutional layers are used to map X to different spaces:

[0035] X1=conv1*X

[0036] X2=conv2*X

[0037] in and Represents two mapped skeleton sequence features, * represents the convolution operator, conv1 and conv2 represent convolution layers, and r = 1. Preferably, the number of convolution kernels in the convolution layer can be adjusted to make r > 1, thereby reducing the number of output feature channels and reducing the amount of computation.

[0038] In order to extract multi-scale short-term motion features, it is necessary to calculate the feature difference between any two skeleton frames. By expanding the dimensions of the skeleton sequence features X1 and X2 respectively, their dimensions become [B, C / r, T, 1, N] and [B, C / r, 1, T, N]. Under the action of the matrix broadcast mechanism, the multi-scale motion activation weight M can be obtained. This process can be expressed as follows

[0039] M=σ(X1-X2)

[0040] in, σ(·) is the hyperbolic tangent (Tanh) activation function.

[0041] M represents the feature difference between any two frames. In order to aggregate the feature differences between one frame and all frames, a global average pooling layer p(·) is used to obtain the aggregated multi-scale motion activation weight M p Since X1 and X2 are channel-reduced, a 1×1 2D convolutional layer (conv3) is needed to convert M p The channel dimension of is restored to the channel dimension of the original node feature X. This process can be expressed as

[0042]

[0043] The purpose of this module is to activate motion-sensitive feature channels, so the multi-scale motion activation weights M p By multiplying the feature X with the skeleton joint point, the motion-sensitive feature channel is activated. When the channel is not reduced, At the same time, another 1×1 2D convolution layer (conv4) is used to update the node features to improve the representation ability of the node feature X. This process can be expressed as

[0044]

[0045] Among them, X m is the updated node feature, and ⊙ represents the dot product operation.

[0046] Although the dot product method can activate motion-sensitive feature channels, it may also suppress other feature channels, such as position information and node type information channels. To solve this problem, the module uses a residual connection to preserve the information of other channels.

[0047] X r =X m +X,

[0048] Among them, X r It is the feature of multi-scale motion activation, Contains multi-scale short-term motion information of skeleton sequences.

[0049] 3.2 Extracting Long-term Temporal Features of Skeleton Joints

[0050] Existing skeleton-based temporal convolution methods can be used to extract long-term temporal features of skeleton joints, such as stacked 1D convolutions with a fixed step size or multi-branch convolutions with varying step sizes. The multi-scale motion activation features described above are input into a temporal convolutional network, resulting in a single-layer output that includes spatial dependencies between skeleton joints, short-term motion features, and long-term temporal features. Figure 3 A schematic diagram of the insertion position of the multi-scale motion activation module is shown, where GC represents graph convolution and TC represents temporal convolution. Existing skeleton-based graph convolutional neural networks mostly construct the network by stacking spatial graph convolution and temporal convolution. This module does not change the dimension of the skeleton features, so it can be inserted without modifying the original network. It obtains multi-scale motion activation weights, activates motion-sensitive feature channels, and extracts multi-scale short-term motion features, thereby improving the recognition performance of the network.

[0051] Step 4: Output the skeleton behavior recognition results.

[0052] The above steps 2 and 3 are operations for one layer of the network. Repeat the above steps until all layers of the graph convolutional neural network complete the network output and obtain the final features of the joints. The final features are pooled and input into the classifier. The classifier maps the final features to behavior categories, and the categories are used as the behavior recognition results of the skeleton.

[0053] The above recognition method is applied to the human action recognition dataset NTU-RGB+D. The action recognition accuracy obtained by this method is shown in Table 1:

[0054]

[0055] Table 1

[0056] Here, Ours represents the method of this embodiment, and the evaluation indicator used is Top 1 accuracy.

[0057] To further verify the effectiveness of the method, this method was applied to another larger-scale human action recognition dataset, NTU-RGB+D120. Compared with other methods, the method in this embodiment achieved the best performance to date. The specific numerical comparison results are shown in Table 2:

[0058]

[0059]

[0060] Table 2

[0061] Among them, Ours represents the method of this embodiment, and the evaluation index used is Top1 accuracy

[0062] A skeleton-based behavior recognition system, used to implement the skeleton-based behavior recognition method, comprising

[0063] The acquisition module is used to obtain the human skeleton coordinates captured by a somatosensory device, such as a somatosensory camera (such as the Microsoft Kinect camera), or to extract the human skeleton coordinates in the video using a human posture estimation method;

[0064] The spatial relationship extraction module is used to extract the spatial features of the skeleton sequence, such as the CTR-GCN network. For scenarios with high real-time requirements, the Shift-GCN network can be selected;

[0065] The motion activation module is used to extract activation features and temporal features. Figure 2 The structure shown;

[0066] The classification module is used to output the skeleton behavior recognition results, such as a classifier.

[0067] A skeleton-based behavior recognition device comprises a processor and a memory, wherein the memory is used to store at least one program, and the processor is used to execute the skeleton-based behavior recognition method according to any one of claims 1 to 6.

[0068] A skeleton-based behavior recognition storage medium is used to store instructions or programs for implementing the skeleton-based behavior recognition method, such as a mobile hard disk or a USB flash drive.

[0069] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A skeleton-based behavior recognition method, characterized in that: include: S1, obtain the coordinates of the human skeleton joints, and obtain a skeleton sequence based on the coordinates of the human skeleton joints in multiple frames; S2, input the skeleton sequence into the graph convolutional neural network, and extract the spatial features of the skeleton sequence through the graph convolutional neural network Where R is the dimension, B is the batch size, C is the channel dimension, T is the length of the skeleton sequence, and N is the number of nodes of the skeleton joint points of a single frame in the skeleton sequence; S3, obtains two features X1 and X2 by convolution operation on feature X. In the process of obtaining features X1 and X2 by convolution operation on feature X, the convolution kernel channel of the convolution operation is controlled to reduce the number of feature X1 and X2 channels. Where r is the channel reduction factor, r is a positive integer, and the dimension of X1 is expanded to [B, C, T, 1, N], and the dimension of X2 is expanded to [B, C, 1, T, N]. Through the matrix broadcast mechanism, the expanded X1 and X2 are adjusted to the same dimension, and the two adjusted features are fused by the formula M = σ(X1-X2), where σ(·) is the hyperbolic tangent activation function, and the feature weight M is obtained. After global averaging of M, the activation weight M is obtained. p , activation weight M p Then, a 2D convolutional layer is used to transform M p Restore to the same channel dimension as feature X, M p Multiply by X to get X m , for X m After the residual connection feature X, the activation feature X is obtained r , that is, X r =X m +X; S4, X r Input the temporal convolutional neural network and get a layer including feature X, feature X m and the network output of temporal features; S5, repeat S2 to S4 until the network output of all layers is obtained as the final feature of the joint point, the final feature is pooled and input into the classifier, the final feature is mapped to the behavior category through the classifier, and the category is used as the behavior recognition result of the skeleton.

2. A skeleton-based behavior recognition system for implementing the skeleton-based behavior recognition method according to claim 1, characterized in that: include The acquisition module is used to obtain the human skeleton coordinates captured by the somatosensory device, or to extract the human skeleton coordinates in the video using the human posture estimation method; Spatial relationship extraction module, used to extract the spatial features of skeleton sequences; Motion activation module, used to extract activation features and temporal features; The classification module is used to output the skeleton behavior recognition results.

3. A skeleton-based behavior recognition device, characterized in that: The invention comprises a processor and a memory, wherein the memory is used to store at least one program, and the processor is used to execute the skeleton-based behavior recognition method according to claim 1.

4. A skeleton-based behavior recognition storage medium, characterized in that: Used to store instructions or programs for implementing the skeleton-based behavior recognition method described in claim 1.

Images