A human interaction behavior recognition method based on spatiotemporal graph convolution

Abstract

The invention discloses a human interaction behavior recognition method based on spatiotemporal graph convolution, which belongs to the field of artificial intelligence. Specifically, the method comprises the following steps: firstly, a video of two-person interaction behavior within a time T is recorded by a depth camera, three-dimensional coordinate data of key points of human skeleton are directly extracted, and the data is regularized into a tensor such as 1×M×C×T×V; a spatiotemporal interaction graph convolution network is initialized, and the input data tensor reaches 6 sub-network layers with the same structure after passing through a normalization layer, and each sub-network layer consists of two single-person feature extraction modules and an interaction feature association module; the single-person feature extraction module comprises a spatial graph convolution module GCN and a temporal convolution module TCN; after 6 layers of spatial graph convolution and temporal convolution, two tensor features R1 and R2 are output, normalized, and then spliced ​​at the channel dimension, and a category number label is obtained after passing through a fully connected layer and a Softmax layer in sequence; finally, a hash table consisting of the category number and the behavior name is queried to obtain the behavior name; the invention effectively improves the aggregation efficiency and recognition accuracy of the interaction feature.

Description

Technical Field

[0001] The present invention belongs to the field of artificial intelligence, and relates to deep learning and graph convolutional neural networks, and specifically is a method for identifying human interaction behaviors based on spatiotemporal graph convolution. Background Art

[0002] Human action recognition refers to the automatic analysis of human actions in unknown video or image sequences [1]. In a broad sense, human action recognition can be divided into five subtasks: untrimmed video classification, trimmed action recognition, temporal action proposal, temporal action localization, and dense captioning events.

[0003] According to the existing research work at home and abroad, interactive behavior recognition can be generally divided into methods based on template matching and methods based on deep learning;

[0004] Representative works of template matching-based methods include Van Gemeren et al. [2] who combined HOG / HOF descriptors to encode feature frames of human body interactions; and Ji et al. [3] who used HMM to model the changes of HOG descriptors over time and distinguish the beginning, middle, and end stages of the interaction behavior.

[0005] Interactive behavior recognition based on deep learning can be mainly divided into methods based on CNN (Convolutional Neural Networks), RNN (Recurrent Neural Networks), and GCN (Graph Convolutional Networks).

[0006] Representative CNN methods include the AlphaAction network proposed by Tang et al. [4], which uses an interactive aggregation structure to uniformly model interactive actions in RGB videos; representative RNN methods include Wang et al. [5], which use LSTM to unify the feature modeling process of single-person actions and interactive actions in videos. The GCN-based method has been the most popular direction of behavior recognition in recent years. ST-GCN was first introduced into the field of human behavior recognition by Yan et al. [6]. The topological structure data constructed from the key points of the human skeleton extracted from the video is sequentially subjected to spatiotemporal graph convolution operations to complete the recognition, surpassing the previous network performance directly based on RGB image analysis.

[0007] Inspired by ST-GCN, some works have also migrated graph convolution operations to interactive behavior recognition. For example, Li et al. [7] proposed K-GCN to embed the knowledge of two-person interaction into the graph convolutional network; Zhu et al. [8] proposed DR-GCN to combine the geometric features and attention features of two-person interaction to construct the two-person interaction topological graph structure.

[0008] Generally speaking, the application of GCN in the field of interactive behavior recognition is not sufficient. The modeling process of interactive behavior usually involves a relatively large number of parameters and calculations, and is lacking in the construction of the spatiotemporal relationship of the interaction.

[0009] [1]Kong Y,Fu Y.Human action recognition and prediction:A survey[J].arXiv preprint arXiv:1806.11230,2018.

[0010] [2]Van Gemeren C, Tan RT, Poppe R, et al.Dyadic interaction detection from pose and flow[C] / / International Workshop on Human Behavior Understanding. Springer, Cham, 2014:101-115.

[0011] [3]Ji X,Wang C,Ju ZA new framework of human interaction recognition based on multiple stage probability fusion[J].Applied Sciences,2017,7(6):567.

[0012] [4]Tang J,Xia J,Mu X,et al.Asynchronous Interaction Aggregation for Action Detection[J].arXiv preprint arXiv:2004.07485,2020.

[0013] [5]Wang M, Ni B, YangX. Recurrent modeling of interaction context for collective activity recognition[C] / / Proceedings of the IEEE Conference onComputer Vision and Pattern Recognition.2017:3048-3056.

[0014] [6] Yan S, Xiong Y, Lin D. Spatial temporal graph convolutional networks for skeleton-based action recognition [J]. arXiv preprint arXiv:1801.07455, 2018.

[0015] [7]Li J,Xie X,Cao Y,et al.Knowledge embedded GCN for skeleton-basedtwo-person interaction recognition[J].Neurocomputing,2021,444:338-348.

[0016] [8]Zhu L,Wan B,Li C,et al.Dyadic relational graph convolutionalnetworks for skeleton-based human interaction recognition[J].PatternRecognition,2021,115:107920. Summary of the invention

[0017] In response to the above problems, the present invention proposes a method for identifying human interactive behavior based on spatiotemporal graph convolution, which can effectively fill the gap in the construction of interactive spatiotemporal relationships through the temporal interaction module and the spatial interaction module, and has the advantages of lower parameter amount and computational complexity.

[0018] The human body interaction behavior recognition method comprises the following specific steps:

[0019] Step 1: Use a depth camera to record a two-person interaction video within a time T as a sample to be identified, and directly extract the three-dimensional coordinate data of the key points of the human skeleton;

[0020] The entire video is required to contain only one interactive behavior and only two people from beginning to end, and the entire bodies of both people are clearly visible.

[0021] Step 2: Regularize the three-dimensional coordinate data of the key points of the skeleton into a tensor of 1×M×C×T×V;

[0022] Where M=2 represents the number of human bodies, C=3 represents the three dimensions of X, Y, and Z of the spatial coordinate data, T represents the length of the video, and V=25 represents the number of key points of the human skeleton;

[0023] Step 3: Initialize the spatiotemporal interaction graph convolutional network, input the data tensor, and infer the category label of the interaction behavior;

[0024] The spatiotemporal interaction graph convolutional network is composed of a normalization layer, 6 sub-network layers with the same structure, and an additional fully connected layer stacked sequentially;

[0025] The specific reasoning process is:

[0026] Step 301: Input data tensor After the normalization layer, normalization is completed in the M×C×V dimension, and the shape of the tensor does not change before and after normalization.

[0027] Step 302: Divide the normalized tensor into two branches, corresponding to the two interacting persons, namely:

[0028] X1=X[:,0,:,:]

[0029] X2=X[:,1,:,:]

[0030] Among them, X1, They represent the data of two people, and their shapes are exactly the same. X[:,0,:,:] represents the operation of taking all values ​​with subscript 0 in the first dimension of tensor X. Similarly, X[:,1,:,:] represents the operation of taking all values ​​with subscript 1 in the first dimension of tensor X.

[0031] Step 303: Send the two-person data X1 and X2 to the current layer of the 6-layer spatiotemporal graph convolution sub-network layer at the same time;

[0032] Each sub-network layer consists of two single-person feature extraction modules and an interactive feature association module; the single-person feature extraction modules include a spatial graph convolution module GCN and a temporal convolution module TCN; the current layer is initially the first layer;

[0033] First, the two-person data X1 and X2 are sent to two single-person feature extraction modules respectively, and the output tensor features after being processed by the spatial graph convolution operation of GCN are Y1 and Y2 respectively;

[0034] It is expressed as:

[0035]

[0036]

[0037] Among them, Y1, is the output tensor feature, C′ is the number of channel dimensions after spatial graph convolution; ReLu represents the linear rectification activation function; BN represents the batch normalization function; is the normalized single-person adjacency matrix, A∈{0,1} V×V It is a matrix used to describe the connection relationship between the key points of a single human skeleton; D is the degree matrix of the single person adjacency matrix A; is a dynamic adjacency matrix; W1, They are the different convolution weights for performing two 1*1 two-dimensional convolution operations on the data X1. Similarly, W3, They are different convolution weights for performing two 1*1 two-dimensional convolution operations on the data X2;

[0038] Then, the output tensor features Y1 and Y2 are sent to their respective TCNs, and temporal convolution operations are performed to output features Z1 and Z2 respectively;

[0039] It is expressed as:

[0040] Z1=ReLu(BN(Y1W5))

[0041] Z2=ReLu(BN(Y2W6))

[0042] Among them, Z1, is the output feature, where T′=T / s, s is the step size, W5 and W6 represent the convolution weights of two different 9*1 two-dimensional convolution operations;

[0043] Finally, the output features Z1 and Z2 are input into the interactive feature association module for temporal and spatial feature association;

[0044] Specifically:

[0045] First, use the S function to perform a downsampling pooling operation on the input features Z1 and Z2 with a sampling rate of r, and the pooled output tensor S1, It is expressed as:

[0046] S1=S(Z1),S2=S(Z2)

[0047] Then, the DTW algorithm is applied to calculate the correlation matrix M of the output features Z1 and Z2 2→1 ∈{0,1} τ×τ ;

[0048] M 2→1 =DTW(mean(S1)),mean(S2))

[0049] The mean function represents the normalization compression of features in the channel dimension (C) and spatial dimension (V). The tensor shape after compression is 1×τ, that is, only the time dimension features are retained;

[0050] For the incidence matrix M 2→1 The element m in i,j (i,j=0,1,...,τ-1), if and only if Z2 i and Z1 j When there is a correlation i,j =1, otherwise m i,j =0.

[0051] Next, the tensor S2 and the correlation matrix M 2→1 Multiplication completes the alignment with tensor S1 in the time dimension;

[0052] It is expressed as:

[0053] S2′=S2M 2→1

[0054] Finally, the aligned features are concatenated in the spatial dimension, and the spatial graph convolution operation of the two-person features is performed to output the graph convolution feature Q;

[0055] It is expressed as:

[0056] P = Concat(S1, S2′)

[0057]

[0058] The concatenation function Concat represents the concatenation of S1 and S2′ in the third dimension. The two-person spatial graph convolution uses the normalized interaction adjacency matrix D I is the degree matrix, A I ∈{0,1} 2V×2VUsed to describe the connection relationship between interactive bone key points; is the convolution weight of the two-person feature space graph convolution, implemented using a 1*1 two-dimensional convolution, and the output

[0059] Interaction adjacency matrix A I The construction process is expressed as:

[0060] A 12 Describes the connection relationship between the skeleton key points of two human bodies, including symmetrical connection and mirror connection. The symmetrical connection is the connection established between the skeleton key points with the same number belonging to the two human bodies, while the mirror connection is the connection established between the skeleton key points of the two human bodies with a mirror relationship.

[0061] After the output feature Q of the above graph convolution is split in the spatial dimension, two tensors of the same shape are obtained. After the linear rectification function ReLu and the batch normalization function BN, they are used as the final outputs Q1 and Q2 of the interactive feature association module, expressed as:

[0062] Q1=ReLu(BN(Q[:,:,:,0]))

[0063] Q2=ReLu(BN(Q[:,:,:,1]))

[0064] Among them, Q1, Q[:,:,:,0] represents the operation of taking all values ​​with subscript 0 in the third dimension of tensor Q. Similarly, Q[:,:,:,1] represents the operation of taking all values ​​with subscript 1 in the third dimension of tensor Q.

[0065] Step 304, return to step 303, and use the output two-component features Q1 and Q2 as the corresponding data of two people, and send them to the next layer of the 6-layer spatiotemporal graph convolution sub-network layer;

[0066] Step 305, repeat until the 6-layer processing is completed, and the two components finally obtained are respectively recorded as R1,

[0067] C out Indicates the size of the first dimension of the output channel dimension;

[0068] T out Indicates the size of the second dimension of the output time dimension;

[0069] Step 306: Normalize the two components R1 and R2 in terms of time dimension and space dimension to obtain And spliced ​​at the channel dimension, and passed through the fully connected layer and the Softmax layer in turn to obtain the inference result;

[0070] It is expressed as:

[0071]

[0072]

[0073] Among them, FC represents the fully connected layer, and Softmax represents the Softmax layer processing; is the inference result, γ is the number of behavior categories;

[0074] Step 307: The inference result is the confidence ranking of all possible behavior classification results of the current input, and the behavior category label corresponding to the maximum confidence value is found, which is the final result label output by the spatiotemporal interaction graph convolutional network;

[0075] Expressed as:

[0076] label = argmax(R)

[0077] Step 4: Use the category number label as the key to query the hash table consisting of all category numbers and behavior category names in the NTU-RGB+D 120 dataset, and obtain the behavior category name;

[0078] The advantages of the present invention are:

[0079] (1) A human interaction behavior recognition method based on spatiotemporal graph convolution uses the DTW (Dynamic Time Wrapping) algorithm to calculate the correlation between the action information of two people in the time dimension, and then completes the alignment and fusion of the action information in the time dimension. It can fill the gap in existing work that does not consider the correlation of time features with low computational complexity;

[0080] (2) A human interaction behavior recognition method based on spatiotemporal graph convolution uses the proposed two-person interaction graph to complete spatial graph convolution. Combined with the DTW algorithm, the spatiotemporal correlation of two-person interaction action information is completed, thereby effectively improving the aggregation efficiency of interaction features and improving recognition accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0081] Figure 1 It is a flowchart of the overall steps of a method for identifying human interaction behavior based on spatiotemporal graph convolution of the present invention;

[0082] Figure 2 This is the structure diagram of the spatiotemporal interactive graph convolutional network described in the present invention;

[0083] Figure 3 This is a structural diagram of a single spatiotemporal graph convolution subnetwork layer of the present invention;

[0084] Figure 4This is a structural diagram of a single person feature extraction module of the present invention;

[0085] Figure 5 This is a schematic diagram of the connection relationship of key points of a single person's skeleton according to the present invention;

[0086] Figure 6 This is a structural diagram of the interactive feature association module of the present invention. DETAILED DESCRIPTION

[0087] The present invention will be further described in detail below with reference to the accompanying drawings and examples.

[0088] The present invention provides a method for human interaction behavior recognition based on spatiotemporal graph convolution. Human interaction behavior recognition is a subclass of trimmed video recognition, that is, a trimmed video containing only one type of interaction action between two people is analyzed to identify the action category to which it belongs.

[0089] like Figure 1 As shown, the human body interaction behavior recognition method comprises the following specific steps:

[0090] Step 1: Use a depth camera to record a two-person interaction video within a time T as a sample to be identified, and directly extract the three-dimensional coordinate data of the key points of the human skeleton;

[0091] The depth camera is a Kinect camera, and the entire video is required to contain only one interactive behavior and only two people from beginning to end, and the whole bodies of the two people are clearly visible.

[0092] Step 2: Regularize the three-dimensional coordinate data of the key points of the skeleton into a tensor of 1×M×C×T×V;

[0093] Where M=2 represents the number of human bodies, C=3 represents the three dimensions of X, Y, and Z of the spatial coordinate data, T represents the length of the video, and V=25 represents the number of key points of the human skeleton;

[0094] Step 3: Initialize the spatiotemporal interaction graph convolutional network, load the pre-trained network parameter weights, input the data tensor, and obtain the category label of the interaction behavior through reasoning;

[0095] like Figure 2 As shown in the figure, the spatiotemporal interaction graph convolutional network is composed of a normalization layer, 6 sub-network layers with the same structure, and an additional fully connected layer stacked in sequence; where "*6" means that the core processing part is composed of 6 spatiotemporal interaction graph convolutional layers with the same structure, and repeated network layers are simplified by "...";

[0096] The specific reasoning process is:

[0097] Step 301: Input data tensor After the one-dimensional batch normalization layer (BatchNormalization Layer), normalization is completed in the M×C×V dimension, and the shape of the tensor does not change before and after normalization.

[0098] Raw data That is, the three-dimensional coordinate sequence of key points of the human skeleton during the interaction between two people within time T;

[0099] Step 302: The normalized tensor is divided into two branches in the first dimension according to the different persons to whom it belongs, corresponding to the two persons interacting, respectively:

[0100] X1=X[:,0,:,:]

[0101] X2=X[:,1,:,:]

[0102] Among them, X1, They represent the data of two people, and their shapes are exactly the same. X[:,0,:,:] represents the operation of taking all values ​​with subscript 0 in the first dimension of tensor X. Similarly, X[:,1,:,:] represents the operation of taking all values ​​with subscript 1 in the first dimension of tensor X.

[0103] Step 303: Send the two-person data X1 and X2 to the current layer of the 6-layer spatiotemporal graph convolution sub-network layer at the same time;

[0104] Each sub-network layer consists of two single-person feature extraction modules and an interactive feature association module; the single-person feature extraction modules include a spatial graph convolution module GCN and a temporal convolution module TCN; the current layer is initially the first layer;

[0105] like Figure 3 As shown in the figure, the flow of feature tensors is as follows: the input features received by the single spatiotemporal graph convolution subnetwork layer of the first layer are X1 and X2, which are processed by two identical single-person feature extraction modules to output Z1 and Z2 respectively, and then the two are input into the interactive feature association module for further operation. The module outputs Q1 and Q2 as the output of the entire layer and then input into the spatiotemporal graph convolution subnetwork layer structure of the second layer, and so on, and finally the final category number label of the current input interactive behavior is obtained;

[0106] Specifically:

[0107] First, the two-person data X1 and X2 are sent to two single-person feature extraction modules respectively, and the output tensor features after being processed by the spatial graph convolution operation of GCN are Y1 and Y2 respectively;

[0108] like Figure 4As shown in the figure, the dotted line divides the whole into two large modules. The small boxes inside represent the small modules. The text in the box represents the name of the module. "1*1 convolution" represents a two-dimensional convolution operation module with a convolution kernel shape of 1*1. Similarly, "9*1 convolution" represents a two-dimensional convolution operation module with a convolution kernel shape of 9*1. BN represents a batch normalization operation module, and ReLu represents a ReLu activation function module. The boxed and B represent the normalized single-person adjacency matrix and the dynamic adjacency matrix automatically learned by the network, respectively; The symbol table represents tensor addition and tensor multiplication operations; the arrows indicate the data flow direction, and the symbols next to the arrows represent the features of each stage. The input of the entire single-person feature extraction module is X, which first enters the spatial graph convolution module. In this module, X is divided into two paths, one of which directly passes through 1*1 convolution, and the other completes the graph convolution operation (same as After vector multiplication, 1*1 convolution is performed. The result tensors of the two operations are added and then batch normalized and ReLu activation function is used to obtain the output Y of the spatial graph convolution module. After 9*1 convolution, batch normalization and ReLu function are used to complete the temporal convolution operation. The output result Z is the output of the entire single-person feature extraction module.

[0109] It follows the sub-network layer design of ST-GCN, which includes a spatial graph convolution module (GCN) and a temporal convolution module (TCN). The spatial graph convolution operation is expressed as:

[0110] It is expressed as:

[0111]

[0112]

[0113] Among them, Y1, is the output tensor feature, C′ is the number of channel dimensions after spatial graph convolution; ReLu represents the linear rectification activation function; BN represents the batch normalization function; is the normalized single-person adjacency matrix A, A∈{0,1} V×V It is used to describe the connection relationship matrix between the key points of a single human skeleton; Figure 5 As shown, the human skeleton connection structure is shown. Circles with numbers are used to represent the key points of the human skeleton. The lines between the circles represent the bones. The numbers in the circles are the serial numbers of the key points of the skeleton, from 1 to 25, which contains a total of 25 key points of the human skeleton.

[0114] For nodes i, j = 0, 1, ..., V-1, if and only if there is a connection between two points (including the connection between the node itself and itself), the element α at the corresponding position in the adjacency matrix i,j =1, otherwise αi,j =0;

[0115] D is the degree matrix of the single-person adjacency matrix A; is a dynamic adjacency matrix, which is driven by data to adaptively adjust the value of the network; W1, They are the different convolution weights for performing two 1*1 two-dimensional convolution operations on the data X1. Similarly, W3, They are different convolution weights for performing two 1*1 two-dimensional convolution operations on the data X2;

[0116] Then, the output tensor features Y1 and Y2 are sent to their respective TCNs, and temporal convolution operations are performed to output features Z1 and Z2 respectively;

[0117] It is expressed as:

[0118] Z1=ReLu(BN(Y1W5))

[0119] Z2=ReLu(BN(Y2W6))

[0120] Among them, Z1, is the output feature, where T′=T / s, s is the step size, and s takes the value of 1 or 2; W5 and W6 represent the convolution weights of two different 9*1 two-dimensional convolution operations respectively;

[0121] Finally, the output features Z1 and Z2 are input into the interactive feature association module for temporal and spatial feature association;

[0122] like Figure 6 As shown in the figure, the internal structure of the interactive feature association module is shown. The entire module is divided into two parts: the time feature association module and the spatial feature association module. Each small box represents the operation performed, and the text in the box is the content of the operation performed. The 1*1 convolution represents a two-dimensional convolution operation with a convolution kernel shape of 1*1. Represents tensor multiplication operation; arrows indicate the direction of data flow; Z1 and Z2 are simultaneously input into the interactive feature association module, and the temporal feature association is completed first. The two are respectively outputted S1 and S2 through the same downsampling pooling, and then the channel space feature is normalized and the DTW algorithm is used to calculate the association matrix M of the two. 2→1 , followed by M 2→1 After multiplying with the S2 tensor, the head of the spatial feature association module completes the splicing of the two tensors with S1 in the spatial dimension to obtain P, P and the normalized interactive adjacency matrix After tensor multiplication, graph convolution is completed through 1*1 convolution, and the shape is reshaped to obtain Q, which is then separated into two tensors of the same shape along the spatial dimension and then subjected to batch normalization and ReLu activation function to obtain the final outputs Q1 and Q2 of the interactive feature association module.

[0123] Specifically:

[0124] First, use the S function to perform a downsampling pooling operation on the input features Z1 and Z2 with a sampling rate of r, and the pooled output tensor S1, It is expressed as:

[0125] S1=S(Z1),S2=S(Z2)

[0126] Then, the DTW algorithm is applied to calculate the correlation matrix M of the output features Z1 and Z2 2→1 ∈{0,1} τ×τ ;

[0127] M 2→1 =DTW(mean(S1)),mean(S2))

[0128] The mean function represents the normalization compression of features in the channel dimension (C) and spatial dimension (V). The tensor shape after compression is 1×τ, that is, only the time dimension features are retained;

[0129] DTW stands for using the DTW algorithm to find the correlation between two tensors in the time dimension. The DTW algorithm itself is a method for measuring the similarity of two time series and is widely used in the field of speech recognition. In order to reduce the computational complexity of the DTW algorithm, this application uses the Manhattan distance as the distance metric of DTW; M 2→1 ∈{0,1} τ×τ The correlation matrix generated by the DTW algorithm, for the element m in the matrix i,j (i,j=0,1,...,τ-1), if and only if Z2 i and Z1 j When there is a correlation i,j =1, otherwise m i,j =0.

[0130] Next, the tensor S2 and the correlation matrix M 2→1 Multiplication completes the alignment with tensor S1 in the time dimension;

[0131] It is expressed as:

[0132] S2′=S2M 2→1

[0133] Finally, the aligned features are concatenated in the spatial dimension, and the spatial graph convolution operation of the two-person features is performed to output the graph convolution feature Q;

[0134] It is expressed as:

[0135] P = Concat(S1, S2′)

[0136]

[0137] The concatenation function Concat represents the concatenation of S1 and S2′ in the third dimension.

[0138] Two-person spatial graph convolution is similar to single-person spatial graph convolution, the only difference is that the normalized interaction adjacency matrix is ​​used D I is the degree matrix, A I ∈{0,1} 2V×2V Used to describe the connection relationship between interactive bone key points; is the convolution weight of the two-person feature space graph convolution, which is implemented using a 1*1 two-dimensional convolution. Note that r here is the sampling rate of the downsampling operation before the previous time association operation. The feature P is After matrix multiplication, the channel dimension is expanded through two-dimensional convolution, and then the Reshape operation is performed to fill the expansion of the channel dimension into the time dimension, so as to ensure the consistency of the shape of the input and output tensors of the interactive feature association module.

[0139] Interaction adjacency matrix A I The construction process is expressed as:

[0140] A I is a block matrix, which is composed of four matrices. A is the aforementioned single-person adjacency matrix, A 12 Describes the connection relationship between the skeletal key points of two human bodies, including symmetrical connection and mirror connection. The symmetrical connection is a connection established between the skeletal key points with the same number belonging to the two human bodies, while the mirror connection is a connection established between the skeletal key points of the two human bodies with a mirror relationship. The design basis of this connection method takes into account the symmetry and mirroring characteristics of two people when interacting.

[0141] In a symmetrical connection, the two human skeleton key points with the same number are connected in correspondence, while in a mirror connection, in addition to the corresponding connection of the skeleton key points of the human torso, the connection between other skeleton key points (limbs) presents a left-right mirror relationship. Assume that the two human bodies are respectively recorded as human body a and human body b, and "ax-by" is used to indicate that the key point x of human body a and the key point y of human body b are connected. Figure 5To illustrate the two connection relationships, all connections established under the symmetric connection include: a1-b1, a2-b2, ... a24-b24, a25-b25; all connections established under the mirror connection are: a1-b1, a2-b2, a3-b3, a4-b4, a21-b21, a5-b9, a9-b5, a6-b10, a10-b6, a11-b7, a7-b11, a12-b8, a8-b12, a25-b23, a23-b25, a24-b22, a22-b24, a17-b13, a13-b17, a18-b14, a14-b18, a19-b15, a15-b19, a20-b16, a16-b20.

[0142] After the output feature Q of the above graph convolution is split in the spatial dimension, two tensors of the same shape are obtained. After the linear rectification function ReLu and the batch normalization function BN, they are used as the final outputs Q1 and Q2 of the interactive feature association module, expressed as:

[0143] Q1=ReLu(BN(Q[:,:,:,0]))

[0144] Q2=ReLu(BN(Q[:,:,:,1]))

[0145] Among them, Q1, Q[:,:,:,0] represents the operation of taking all values ​​with subscript 0 in the third dimension of tensor Q. Similarly, Q[:,:,:,1] represents the operation of taking all values ​​with subscript 1 in the third dimension of tensor Q.

[0146] So far, this sub-network layer has completed the single-person feature extraction and interactive feature association of the input features.

[0147] Step 304, return to step 303, and use the output two-component features Q1 and Q2 as the corresponding data of two people, and send them to the next layer of the 6-layer spatiotemporal graph convolution sub-network layer;

[0148] Step 305, repeat until the 6-layer processing is completed, and the two components finally obtained are respectively recorded as R1,

[0149] C out Indicates the size of the first dimension of the output channel dimension;

[0150] T out Indicates the size of the second dimension of the output time dimension;

[0151] Step 306: Normalize the two components R1 and R2 in terms of time dimension and space dimension to obtain And concatenate at the first dimension (channel dimension), and pass through the fully connected layer FC and the Softmax layer in sequence to obtain the inference result;

[0152] It is expressed as:

[0153]

[0154]

[0155] Among them, FC represents the fully connected layer, and Softmax represents the Softmax layer processing; is the inference result, γ is the number of behavior categories;

[0156] Step 307: The inference result is the confidence ranking of all possible interaction behavior classification results of the current input, and the behavior category label corresponding to the maximum confidence value is found, which is the final category label output by the spatiotemporal interaction graph convolutional network;

[0157] Expressed as:

[0158] label = argmax(R)

[0159] Where argmax represents the function of finding the maximum value of the independent variable;

[0160] In actual implementation, the parameter settings of each sub-network layer are different, as shown in Table 1. The number of input channels of the last fully connected layer of the network is set to 512, and the number of output channels is set to γ.

[0161] Table 1

[0162]

[0163] Step 4: Use the category number label as the key to query the hash table consisting of all category numbers and behavior category names in the NTU-RGB+D 120 dataset to obtain the behavior category name.

Claims

1. A human interaction behavior recognition method based on spatiotemporal graph convolution, characterized in that: The specific steps are as follows: First, a two-person interaction video recorded within a time T by a depth camera is used as a sample to be identified, and the three-dimensional coordinate data of the key points of the human skeleton are directly extracted; the three-dimensional coordinate data of the key points of the skeleton are regularized into a tensor such as 1×M×C×T×V; Where M represents the number of human bodies, C represents the three dimensions of X, Y, and Z of the spatial coordinate data, T represents the length of the video, and V represents the number of key points of the human skeleton; Then, initialize the spatiotemporal interaction graph convolutional network consisting of a normalization layer, 6 sub-network layers with the same structure, and an additional fully connected layer, input the data tensor, and infer the category label of the interaction behavior; The reasoning process of the spatiotemporal interactive graph convolutional network is as follows: Step 301: Input data tensor After the normalization layer, normalization is completed in the M×C×V dimension, and the shape of the tensor does not change before and after normalization; Step 302: Divide the normalized tensor into two branches, corresponding to the two interacting persons, namely: X1=X[:,0,:,:] X2=X[:,1,:,:] in They represent the data of two people, and their shapes are exactly the same; X[:,0,:,:] represents the operation of taking all values ​​with subscript 0 in the first dimension of tensor X, and similarly X[:,1,:,:] represents the operation of taking all values ​​with subscript 1 in the first dimension of tensor X; Step 303: Send the two-person data X1 and X2 to the current layer of the 6-layer spatiotemporal graph convolution sub-network layer at the same time; Each sub-network layer consists of two single-person feature extraction modules and an interactive feature association module; the single-person feature extraction modules include a spatial graph convolution module GCN and a temporal convolution module TCN; the current layer is initially the first layer; First, the two-person data X1 and X2 are sent to two single-person feature extraction modules respectively, and the output tensor features after being processed by the spatial graph convolution operation of GCN are Y1 and Y2 respectively; It is expressed as: in is the output tensor feature, C′ is the number of channel dimensions after spatial graph convolution; ReLu represents the linear rectification activation function; BN represents the batch normalization function; is the normalized single-person adjacency matrix, A∈{0,1} V×V It is a matrix used to describe the connection relationship between the key points of a single human skeleton; D is the degree matrix of the single person adjacency matrix A; is a dynamic adjacency matrix; They are different convolution weights for performing two 1*1 two-dimensional convolution operations on data X1. They are different convolution weights for performing two 1*1 two-dimensional convolution operations on the data X2; Then, the output tensor features Y1 and Y2 are sent to their respective TCNs, and temporal convolution operations are performed to output features Z1 and Z2 respectively; It is expressed as: Z1=ReLu(BN(Y1W5)) Z2=ReLu(BN(Y2W6)) in is the output feature, where T′=T / s, s is the step size, W5 and W6 represent the convolution weights of two different 9*1 two-dimensional convolution operations; Finally, the output features Z1 and Z2 are input into the interactive feature association module for temporal and spatial feature association; Specifically: First, use the S function to perform a downsampling pooling operation on the input features Z1 and Z2 with a sampling rate of r, and the pooled output tensor It is expressed as: S1=S(Z1),S2=S(Z2) Then, the DTW algorithm is applied to calculate the correlation matrix M of the output features Z1 and Z2 2→1 ∈{0,1} τ×τ ; M 2→1 =DTW(mean(S1)),mean(S2)) The mean function represents the normalization compression of features in the channel dimension (C) and spatial dimension (V). The tensor shape after compression is 1×τ, that is, only the time dimension features are retained; For the incidence matrix M 2→1 The element m in i,j (i,j=0,1,...,τ-1), if and only if Z2 i and Z1 j When there is a correlation i,j =1, otherwise m i,j =0; Next, the tensor S2 and the correlation matrix M 2→1 Multiplication completes the alignment with tensor S1 in the time dimension; It is expressed as: <h2 style=";text-align:left;direction:ltr">S2′ = S2M<h2 style=";text-align:left;direction:ltr"> 2→1 Finally, the aligned features are concatenated in the spatial dimension, and the spatial graph convolution operation of the two-person features is performed to output the graph convolution feature Q; It is expressed as: P = Concat(S1, S2′) The concatenation function Concat represents the concatenation of S1 and S2′ in the third dimension. The two-person spatial graph convolution uses the normalized interaction adjacency matrix D I is the degree matrix, A I ∈{0,1} 2V×2V Used to describe the connection relationship between interactive bone key points; is the convolution weight of the two-person feature space graph convolution, implemented using a 1*1 two-dimensional convolution, and the output After the output feature Q of the above graph convolution is split in the spatial dimension, two tensors of the same shape are obtained. After the linear rectification function ReLu and the batch normalization function BN, they are used as the final outputs Q1 and Q2 of the interactive feature association module, expressed as: Q1=ReLu(BN(Q[:,:,:,0])) Q2=ReLu(BN(Q[:,:,:,1])) in Q[:,:,:,0] represents the operation of taking all values ​​with subscript 0 in the third dimension of tensor Q. Similarly, Q[:,:,:,1] represents the operation of taking all values ​​with subscript 1 in the third dimension of tensor Q. Step 304, return to step 303, and use the output two-component features Q1 and Q2 as the corresponding data of two people, and send them to the next layer of the 6-layer spatiotemporal graph convolution sub-network layer; Step 305, repeat until the processing of 6 layers is completed, and the two components finally obtained are recorded as C out Indicates the size of the first dimension of the output channel dimension; T out Indicates the size of the second dimension of the output time dimension; Step 306: Normalize the two components R1 and R2 in terms of time dimension and space dimension to obtain And spliced ​​at the channel dimension, and passed through the fully connected layer and the Softmax layer in turn to obtain the inference result; It is expressed as: Among them, FC represents the fully connected layer, and Softmax represents the Softmax layer processing; is the inference result, γ is the number of behavior categories; Step 307: The inference result is the confidence ranking of all possible behavior classification results of the current input, and the behavior category label corresponding to the maximum confidence value is found, which is the final result label output by the spatiotemporal interaction graph convolutional network; Expressed as: label = argmax(R) Where argmax represents the function of finding the maximum value of the independent variable; Finally, use the category number label as the key to query the hash table consisting of all category numbers and behavior category names in the NTU-RGB+D 120 dataset to obtain the behavior category name.

2. The method for identifying human interaction behavior based on spatiotemporal graph convolution according to claim 1, characterized in that: The two-person interaction video is required to contain only one interactive behavior in the entire video and only contain two people from beginning to end, and the entire bodies of the two people are clearly visible.

3. The method for human interaction behavior recognition based on spatiotemporal graph convolution as claimed in claim 1, characterized in that: In step 303, the interactive adjacency matrix A I The construction process is expressed as: A 12 Describes the connection relationship between the skeleton key points of two human bodies, including symmetrical connection and mirror connection. The symmetrical connection is the connection established between the skeleton key points with the same number belonging to the two human bodies, while the mirror connection is the connection established between the skeleton key points of the two human bodies with a mirror relationship.

Images