Action recognition method and system based on fusion graph convolutional network and Transformer network

Abstract

A method and system for action recognition based on a fusion graph convolutional network and a Transformer network, belonging to the technical fields of computer vision, etc., solves the problem that the prior art only shallowly uses graph convolutional networks to extract spatial features, and its local spatiotemporal feature modeling capability is poor. The present invention preprocesses the acquired human skeleton sequence data, and introduces joint velocity information to obtain a human skeleton sequence with joint velocity information; constructs a spatiotemporal skeleton graph based on the human skeleton sequence with joint velocity information; constructs an adaptive spatiotemporal graph convolution unit module of the spatiotemporal skeleton graph and stacks it, and obtains a spatiotemporal graph convolution network after stacking; fuses the spatiotemporal graph convolution network and the visual Transformer network through a fusion layer to obtain an action recognition model and trains it; after processing the human action skeleton sequence to be predicted, it is sent to the trained action recognition model for prediction, and the action category is predicted. The present invention is used for action recognition.

Description

Technical Field

[0001] An action recognition method and system based on a fusion graph convolutional network and a Transformer network are used for action recognition, belonging to the technical fields of computer vision, action recognition and deep learning. Background Art

[0002] Action recognition technology is one of the most basic and core research areas in the field of computer vision, mainly for classifying human actions. Action recognition technology has been widely studied, developed and applied in many fields such as academia and industry. Since RGB data itself has weak anti-interference ability, problems such as visual occlusion, background interference and illumination changes have always restricted the development of action recognition based on RGB video. Skeleton data removes clothing and body shape information, and focuses on the spatial position information of human joints. Compared with image information, it has more advanced and accurate information expression capabilities and strong adaptability. Because its data itself is not affected by viewpoint changes, illumination changes and other issues, it has good robustness in action recognition tasks.

[0003] The skeleton sequence contains the spatial position information of the human skeleton joints. By describing and modeling the spatial position changes of the human skeleton joints between frames, the spatiotemporal change characteristics of the skeleton joints during the action are extracted to achieve action recognition. With the development of deep learning technology, more and more network models in deep learning are used in action recognition tasks based on skeleton data. Common algorithms can be divided into the following three categories: methods based on recurrent neural networks (RNN), conventional neural networks (CNN) and graph convolutional neural networks (GCN). The GCN-based method processes skeleton data according to the topological graph structure and has become the mainstream.

[0004] The RNN-based method in the prior art processes the action sequence according to the time vector, which loses the spatial structure information of the joint points, resulting in poor accuracy in judging the action category. The CNN-based method processes the skeleton sequence in the form of a pseudo-image, and lacks the modeling expression of the connection between the joints, resulting in poor accuracy in judging the action category. In ST-GCN, the human skeleton is constructed into a fixed-structure spatiotemporal skeleton graph according to the physiological structure. Due to the fixed graph structure, the node information can only be transmitted along a fixed path. Nodes with a long distance can only transmit and interact information through high-order convolution, which easily leads to the lack of extraction of the spatial structure features of the skeleton, and the action features cannot be accurately represented, resulting in poor model robustness. Both the ST-GCN and 2s-AGCN methods use a temporal convolutional network to perform temporal convolution within a local time range in the entire model. Extracting the time dimension information based on the temporal convolutional network will not fully represent the temporal dynamic information of the entire action process. The method has the problem of high computational complexity and incomplete long-term feature extraction, resulting in low model accuracy.

[0005] CN113111760A-Lightweight graph convolution human skeleton action recognition method based on channel attention uses the adjacency matrix and the residual GCN network that introduces the channel attention mechanism to extract the spatial features of the skeleton sequence, and then extracts the temporal features of the skeleton sequence after pooling and aggregating the spatial features to complete the action category judgment, but there are the following technical problems:

[0006] 1. Only shallow graph convolutional networks are used to extract spatial features, and their local spatiotemporal feature modeling capabilities are poor, resulting in poor accuracy in action category judgment;

[0007] 2. The adjacency matrix is ​​obtained by using the inner product of the double-layer fully connected layer to encode the skeleton sequence information, without considering the physiological connection relationship of the skeleton, which makes the graph convolution layer weak in modeling the skeleton graph, resulting in poor accuracy in action category judgment;

[0008] 3. The first-order CNN network is used to extract the temporal features of the skeleton sequence, but has poor modeling capabilities for long-range temporal features, resulting in poor accuracy in action category judgment. Summary of the invention

[0009] In response to the above research problems, the purpose of the present invention is to provide an action recognition method and system based on the fusion of graph convolutional network and Transformer network, so as to solve the problem that the prior art only shallowly uses graph convolutional network to extract spatial features, and its local spatiotemporal feature modeling ability is poor, resulting in poor accuracy of action category judgment.

[0010] In order to achieve the above object, the present invention adopts the following technical solution:

[0011] An action recognition method based on a fusion graph convolutional network and a Transformer network includes the following steps:

[0012] S1: preprocessing the acquired human skeleton sequence data, introducing joint velocity information based on the result obtained after the preprocessing, and obtaining a human skeleton sequence with the joint velocity information introduced, wherein the human skeleton sequence data is a frame sequence of the skeleton;

[0013] S2: Constructing a spatiotemporal skeleton graph based on the human skeleton sequence with joint velocity information;

[0014] S3: construct an adaptive spatiotemporal graph convolution unit module of the spatiotemporal skeleton graph and stack them to obtain a spatiotemporal graph convolution network for extracting local spatiotemporal features of the spatiotemporal skeleton graph;

[0015] S4: The spatiotemporal graph convolutional network and the visual Transformer network that extracts long-range temporal features from the pseudo image feature map and performs action classification are fused through a fusion layer, and the fused action recognition model is trained to obtain a trained action recognition model, wherein the fusion layer obtains the pseudo image feature map based on local spatiotemporal features;

[0016] S5: The human action skeleton sequence to be predicted is processed into a temporal skeleton graph and then sent to the trained action recognition model for prediction to obtain the action category.

[0017] Further, the specific steps of step S1 are:

[0018] S1.1: Normalize the acquired human skeleton sequence based on the BN layer;

[0019] S1.2: Randomly crop the normalized human skeleton sequence to a length greater than 64 frames;

[0020] S1.3: using array sampling operations to perform up- and down-sampling operations on the spatial position array of the cropped human skeleton sequence in the time dimension, scaling the cropped human skeleton sequence to a length of 64 frames, thereby obtaining a scaled human skeleton sequence;

[0021] S1.4: Based on the scaled human skeleton sequence, the velocity information corresponding to each joint point is obtained by performing differential calculation on the previous and next frames. For the i-th joint point v in frame t, ti And the joint point v corresponding to this joint point in frame t+1 (t+1)i , and their spatial position information are respectively (x ti ,y ti , z ti ) and (x (t+1)i ,y (t+1)i , z (t+l)i), then for the joint point v in frame t ti , its speed information is (x (t+1)i -x ti ,y (t+1)i -y ti , z (t+1)i -z ti ), the position information and speed information are spliced ​​together to obtain the preprocessed human skeleton sequence, and the spliced ​​joint point v ti The information can be expressed as: (x (t+1)i ,y (t+1)i , z (t+1)i , x (t+1)i -x ti ,y (t+1)i -y ti , z (t+1)i -z ti ), x, y, z represent three-dimensional coordinate points.

[0022] Further, the specific steps of step S2 are:

[0023] S2.1: The preprocessed human skeleton sequence is represented by an undirected graph G = (V, E), wherein the human skeleton sequence has T frames and N joints, V in the undirected graph G represents the set of joints, and E represents the set of edges, i.e., the set of bone connections;

[0024] S2.2: V = {v ti |t=1,L,T,i=1,L,N} represents all the related nodes in the undirected graph G, t represents the frame index number, i represents the joint index number, T represents the total number of frames, and N represents the total number of joints;

[0025] S2.3: The set of edges E is given by the set of spatial edges E s and the set of time edges Et, that is, E=(E s , E t ), subset E s = {v ti v tj |(i, j)∈H} is the connection between joints in each frame space dimension, which belongs to the intra-frame connection, where H is the set of joint connections of the human body in space, and the subset E t = {v ti v (t+1)i} is the connection between the same joint point between the previous and next frames, which belongs to the inter-frame connection;

[0026] S2.4: Taking the point p in space as the center point, define the neighboring point set B(v ti )={v tj |d(v tj , v ti)≤D},d(v tj , v ti ) represents the jth joint point v in frame t tj To the i-th joint point v ti The shortest distance, D is a given threshold, which is 1. The neighboring point set of each joint point is marked to define the index order in the graph convolution process, that is, the joint point v ti The neighboring point set B(v ti ) is divided into three subsets according to the root node subset, the centripetal point subset, and the centrifugal point subset. The division rules and the defined index order are as follows:

[0027]

[0028] Among them, r i is the distance from joint point i to the center of gravity of the skeleton. Joint point i is the central joint point of the neighboring point set. The center of gravity of the skeleton is the coordinate position obtained by taking the average of the coordinate positions of all joint points of the entire human skeleton in a single frame. j is the distance from joint point j to the center of gravity of the skeleton;

[0029] After partitioning and defining the index order, a spatiotemporal skeleton graph is obtained.

[0030] Further, the specific steps of step S3 are:

[0031] S3.1: Based on the three subsets, a single-layer adaptive graph convolution layer is designed using an adaptive graph structure. The calculation formula is as follows:

[0032]

[0033] Among them, f in is the input of the spatial graph convolution layer, which refers to a single-layer adaptive graph convolution layer. out is the output of the spatial graph convolutional layer, W k is the weight matrix of the kth trainable subset. The number of partitions K of the root node subset, the centripetal point subset, and the centrifugal point subset is 3, which is the normalized adjacency matrix of the spatiotemporal skeleton graph. Divided into 3 partsA k , A1 is the unit matrix I, It is divided into two different directed graphs A2 and A3, A2 points to the center of gravity of the human skeleton, and A3 is far away from the center of gravity of the human skeleton. Among them, the degree matrix For the same value k, the adjacency matrix B k A k Initialize the adjacency matrix parameters and initialize the adjacency matrix B during the action recognition model training process. k The parameters are updated, and the adjacency matrix C kRelying on different human skeleton sequence sample data to generate graph structure information, the normalized Gaussian embedding function is used to capture the interaction between two joints to obtain the correlation dependency between the two joints. Since the normalized Gaussian embedding is equivalent to the softmax operation, the adjacency matrix C k The calculation formula is as follows:

[0034]

[0035] in, represents the inversion of the parameter matrix of the trainable linear layer θ of the kth subset, W φk represents the parameter matrix of the trainable linear layer φ of the kth subset, With W φk For use with f in Multiplication achieves data transformation.

[0036] S3.2: Constructing the residual channel attention layer The output features of the spatial graph convolution layer are fed into the introduced residual channel attention layer, that is, the features f output by the spatial graph convolution network are out Implementing the extrusion operation F sq (·), feature f out The number of channels is c-dimensional, and a c-dimensional vector S is obtained. The vector S is sent to two fully connected layers for excitation operation F. ex (·), get a new c-dimensional weight vector EE, and combine the vector EE with the spatial graph convolutional layer output feature f out After multiplication, it is output with the spatial graph convolution layer feature f out Perform residual connection and add to get the final output feature f;

[0037] S3.3: Use the traditional convolutional layer as the temporal convolutional layer to extract the local temporal features of the human skeleton sequence, and the convolution kernel size is 1×Γ, where Γ is the number of frames spanned by the temporal convolution kernel;

[0038] S3.4: The adaptive graph convolution layer, batch normalization layer, residual channel attention layer, ReLu activation function layer, temporal convolution layer, batch normalization layer and ReLu activation function layer are sequentially connected to form an adaptive spatiotemporal graph convolution unit module;

[0039] S3.5: Connect three adaptive spatiotemporal graph convolution unit modules in sequence, and set the temporal convolution step of each temporal convolution layer to 1, to obtain an adaptive spatiotemporal graph convolution unit module based on the stack, that is, the spatiotemporal graph convolution network.

[0040] Furthermore, the excitation operation F in step S3.2 ex(·) Specifically, the C / r-dimensional vector is obtained through the first fully connected layer, and then ReLu activation is performed. Then the second fully connected layer is performed to convert the C / r-dimensional vector back to a C-dimensional vector, and Sigmoid activation is performed again to make the value between 0 and 1, and finally the c-dimensional weight vector EE is obtained.

[0041] Furthermore, in step S4, the specific implementation steps of the fusion layer are:

[0042] S4.1: In the spatial dimension, that is, in the node dimension of the feature tensor, global average pooling is performed to aggregate the local spatiotemporal features in the spatial dimension of the joint points, so that the action features become time series features;

[0043] S4.2: The time series features are transformed into an array in chronological order from left to right and from top to bottom to obtain a two-dimensional representation of the time series features, which remains unchanged in the channel direction, and finally a pseudo image feature map is generated.

[0044] An action recognition system based on a fusion graph convolutional network and a Transformer network, comprising:

[0045] Data processing module: preprocessing the acquired human skeleton sequence data, introducing joint velocity information based on the result obtained after preprocessing, and obtaining a human skeleton sequence with introduced joint velocity information, wherein the human skeleton sequence data is a frame sequence of the skeleton;

[0046] Spatiotemporal skeleton graph construction module: constructs a spatiotemporal skeleton graph based on the human skeleton sequence with joint velocity information;

[0047] Spatiotemporal graph convolutional network construction module: constructs the adaptive spatiotemporal graph convolution unit module of the spatiotemporal skeleton graph and stacks it. After stacking, a spatiotemporal graph convolutional network is obtained to extract the local spatiotemporal features of the spatiotemporal skeleton graph;

[0048] Fusion module: The spatiotemporal graph convolutional network and the visual Transformer network that extracts long-range temporal features from the pseudo image feature map and performs action classification are fused through the fusion layer, and the action recognition model obtained after the fusion is trained to obtain a trained action recognition model. The fusion layer obtains the pseudo image feature map based on the local spatiotemporal features.

[0049] Prediction module: The human action skeleton sequence to be predicted is processed into a temporal skeleton graph and then sent to the trained action recognition model for prediction to obtain the action category.

[0050] Furthermore, the specific implementation steps of the step data processing module are:

[0051] S1.1: Normalize the acquired human skeleton sequence based on the BN layer;

[0052] S1.2: Randomly crop the normalized human skeleton sequence to a length greater than 64 frames;

[0053] S1.3: using array sampling operations to perform up- and down-sampling operations on the spatial position array of the cropped human skeleton sequence in the time dimension, scaling the cropped human skeleton sequence to a length of 64 frames, thereby obtaining a scaled human skeleton sequence;

[0054] S1.4: Based on the scaled human skeleton sequence, the velocity information corresponding to each joint point is obtained by performing differential calculation on the previous and next frames. For the i-th joint point v in frame t, ti And the joint point v corresponding to this joint point in frame t+1 (t+1)i , and their spatial position information are respectively (x ti ,y ti , z ti ) and (x (t+1)i ,y (t+1)i , z (t+1)i ), then for the joint point v in frame t ti , its speed information is (x (t+1)i -x ti ,y (+1)i -y ti , z (t+1)i -z ti ), the position information and speed information are spliced ​​together to obtain the preprocessed human skeleton sequence, and the spliced ​​joint point v ti The information can be expressed as: (x (t+1)i ,y (t+1)i , z (t+1)i , x (t+1)i -x ti ,y (t+l)i -y ti , z (t+1)i -z ti ), x, y, z represent three-dimensional coordinate points.

[0055] Furthermore, the specific implementation steps of the spatiotemporal skeleton graph construction module are:

[0056] S2.1: The preprocessed human skeleton sequence is represented by an undirected graph G = (V, E), wherein the human skeleton sequence has T frames and N joints, V in the undirected graph G represents the set of joints, and E represents the set of edges, i.e., the set of bone connections;

[0057] S2.2: V = {v ti|t=1,L,T,i=1,L,N} represents all the related nodes in the undirected graph G, t represents the frame index number, i represents the joint index number, T represents the total number of frames, and N represents the total number of joints;

[0058] S2.3: The set of edges E is given by the set of spatial edges E s and the set E of time edges t The two parts together constitute, that is, E=(E s , E t ), subset E s = {v ti v tj |(i, j)∈H} is the connection between joints in each frame space dimension, which belongs to the intra-frame connection, where H is the set of joint connections of the human body in space, and the subset E t = {v ti v (t+l)i} is the connection between the same joint point between the previous and next frames, which belongs to the inter-frame connection;

[0059] S2.4: Taking the point p in space as the center point, define the neighboring point set B(v ti )={v tj |d(v tj , v ti )≤D},d(v ti , v ti ) represents the jth joint point v in frame t tj To the i-th joint point v ti The shortest distance, D is a given threshold, which is 1. The neighboring point set of each joint point is marked to define the index order in the graph convolution process, that is, the joint point v ti The neighboring point set B(v ti ) is divided into three subsets according to the root node subset, the centripetal point subset, and the centrifugal point subset. The division rules and the defined index order are as follows:

[0060]

[0061] Among them, r i is the distance from joint point i to the center of gravity of the skeleton. Joint point i is the central joint point of the neighboring point set. The center of gravity of the skeleton is the coordinate position obtained by taking the average of the coordinate positions of all joint points of the entire human skeleton in a single frame. j is the distance from joint point j to the center of gravity of the skeleton;

[0062] After partitioning and defining the index order, a spatiotemporal skeleton graph is obtained;

[0063] The specific implementation steps of the spatiotemporal graph convolutional network construction module are:

[0064] S3.1: Based on the three subsets, a single-layer adaptive graph convolution layer is designed using an adaptive graph structure. The calculation formula is as follows:

[0065]

[0066] Among them, f in is the input of the spatial graph convolution layer, which refers to a single-layer adaptive graph convolution layer. out is the output of the spatial graph convolutional layer, W k is the weight matrix of the kth trainable subset. The number of partitions K of the root node subset, the centripetal point subset, and the centrifugal point subset is 3, which is the normalized adjacency matrix of the spatiotemporal skeleton graph. Divided into 3 partsA k , A1 is the unit matrix I, It is divided into two different directed graphs A2 and A3, A2 points to the center of gravity of the human skeleton, and A3 is far away from the center of gravity of the human skeleton. Among them, the degree matrix For the same value k, the adjacency matrix B k A k Initialize the adjacency matrix parameters and initialize the adjacency matrix B during the action recognition model training process. k The parameters are updated, and the adjacency matrix C k Based on the graph structure information generated by different human skeleton sequence sample data, the normalized Gaussian embedding function is used to capture the interaction between two joints, and the correlation dependency between the two joints is obtained. Since the normalized Gaussian embedding is equivalent to the softmax operation, the adjacency matrix C k The calculation formula is as follows:

[0067]

[0068] in, represents the inversion of the parameter matrix of the trainable linear layer θ of the kth subset, W φk represents the parameter matrix of the trainable linear layer φ of the kth subset, With W φk For use with f in Multiplication achieves data transformation.

[0069] S3.2: Constructing the residual channel attention layer The output features of the spatial graph convolution layer are fed into the introduced residual channel attention layer, that is, the features f output by the spatial graph convolution network are out Implementing the extrusion operation F sq (·), feature f out The number of channels is c-dimensional, and a c-dimensional vector S is obtained. The vector S is sent to two fully connected layers for excitation operation F. ex(·), get a new c-dimensional weight vector EE, and combine the vector EE with the spatial graph convolutional layer output feature f out After multiplication, it is output with the spatial graph convolution layer feature f out Perform residual connection and add to get the final output feature f; among them, the excitation operation F ex (·) Specifically, the C / r-dimensional vector is obtained through the first fully connected layer, and then ReLu activation is performed. Then, the second fully connected layer is performed to convert the C / r-dimensional vector back to a C-dimensional vector, and Sigmoid activation is performed again to make the value between 0 and 1, and finally the c-dimensional weight vector EE is obtained;

[0070] S3.3: Use the traditional convolutional layer as the temporal convolutional layer to extract the local temporal features of the human skeleton sequence, and the convolution kernel size is 1×Γ, where Γ is the number of frames spanned by the temporal convolution kernel;

[0071] S3.4: The adaptive graph convolution layer, batch normalization layer, residual channel attention layer, ReLu activation function layer, temporal convolution layer, batch normalization layer and ReLu activation function layer are sequentially connected to form an adaptive spatiotemporal graph convolution unit module;

[0072] S3.5: connect three adaptive spatiotemporal graph convolution unit modules in sequence, and set the temporal convolution step of each temporal convolution layer to 1, so as to obtain an adaptive spatiotemporal graph convolution unit module based on stacking, i.e., a spatiotemporal graph convolution network;

[0073] Furthermore, in the fusion module, the specific implementation steps of the fusion layer are:

[0074] S4.1: In the spatial dimension, that is, in the node dimension of the feature tensor, global average pooling is performed to aggregate the local spatiotemporal features in the spatial dimension of the joint points, so that the action features become time series features;

[0075] S4.2: The time series features are transformed into an array in chronological order from left to right and from top to bottom to obtain a two-dimensional representation of the time series features, which remains unchanged in the channel direction, and finally a pseudo image feature map is generated.

[0076] Compared with the prior art, the present invention has the following beneficial effects:

[0077] First, the present invention learns and updates the adjacency matrix of the fixed structure designed according to the physiological structure of the human body during the network model training process (i.e., the adjacency matrix B k It is learned and updated during the network training process), and at the same time, for each sample data, the connection relationship and connection strength between nodes are adaptively calculated based on the node characteristics (adjacency matrix C k is to use input f inThe two linear layers are learned during the training process, and the connection relationship and connection strength are adaptively adjusted for each sample data. The two adjacency matrices are combined (i.e., the adjacency matrix B k and the adjacency matrix C k The combination of ( ) can realize personalized spatiotemporal feature extraction of each sample and improve the robustness of action recognition based on skeleton data.

[0078] 2. The present invention uses the Transformer framework to extract long-range temporal features. By rearranging the time series features, the continuous and discontinuous moments can achieve feature interaction, and the long-range temporal features of the action sequence are extracted as a whole. The general visual Transformer framework for image classification can be used for replacement, and the method is simpler to use.

[0079] 3. The present invention uses a multi-layer adaptive spatiotemporal graph convolution unit to extract the local spatiotemporal features of all joints in the human skeleton sequence, and further pools and compresses (aggregates) the local spatiotemporal features in the node dimension and sends them to the Transformer framework for long-range time feature extraction. The whole method achieves the extraction of local spatiotemporal features and long-range time features with a smaller amount of computation, making the model more lightweight while having better accuracy and robustness. BRIEF DESCRIPTION OF THE DRAWINGS

[0080] Figure 1 is a flow chart of the present invention;

[0081] Figure 2 It is a spatiotemporal skeleton graph constructed based on the human skeleton sequence of the present invention;

[0082] Figure 3 It is a schematic diagram of subset partitioning of the neighboring point set of the joint point of the spatiotemporal skeleton graph in the present invention;

[0083] Figure 4 is a schematic diagram of the adaptive spatial graph convolution layer in the present invention, in which res 1×1 represents the use of 1×1 convolution for residual connection;

[0084] Figure 5 : is a schematic diagram of the residual channel attention layer in the present invention, in which f out The C, W, and H in the table represent f out The C-dimensional vector, W-dimensional vector, and H-dimensional vector of f, where C, W, and H in f represent the C-dimensional vector, W-dimensional vector, and H-dimensional vector of f respectively;

[0085] Figure 6: is a schematic diagram of the adaptive spatiotemporal graph convolution unit module in the present invention, wherein GCN represents an adaptive graph convolution layer, BN represents a batch normalization layer, RCAB represents a residual channel attention layer, ReLU represents a ReLu activation function layer, and TCN represents a temporal convolution layer;

[0086] Figure 7 It is a structural schematic diagram of the action recognition method of the present invention that integrates the spatiotemporal graph convolutional network and the visual Transformer network, wherein N represents input data, BN represents BN layer, GCN represents adaptive graph convolution layer, TCN represents temporal convolution layer, Channel Attention represents residual channel attention layer, GAP represents global average pooling, Transformer represents visual Transformer network, FC represents fully connected layer, and OUT is the output result;

[0087] Figure 8 : is a schematic diagram of the structure of the ViT model described in the present invention, wherein Linear Projection of Flatten Patches represents linear projection of flattened image blocks, Patch+Positional Embedding represents block and position encoding of image blocks, Transformer Encoder represents an encoder, and MLP Head represents a multi-layer perceptron;

[0088] Fig. 9 This is a schematic diagram of the structure of the Swin Transformer model, where Images represents the input image, 3×H×W represents the image size, Patch Partition represents the image block division, and Linear Embedding represents the calculation of the linear embedding vector for the image block. All represent feature map sizes, ×2 and ×6 represent stacking of different numbers of modules, Swin Transformer Block represents encoder blocks, Stage1, Stage2, Stage3, and Stage4 represent stages 1, 2, 3, and 4, respectively;

[0089] Fig.10 It is a structural diagram of the action recognition method using the Swin Transformer structure of the Transformer network in the method disclosed by the present invention. Reshape represents the shape transformation of the feature map size, Pseudo lmages represents the pseudo image feature map, 64×H×W, C×H×W, All represent feature map sizes, ×n1, ×n2, and ×n3 represent stacking of different numbers of modules, Skeleton Sequence represents action sequence, Patch merging represents image block merging, and Block represents image block;

[0090] Fig.11 is the confusion matrix of the test results of the present invention under the x-view test benchmark on the NTU RGB+D dataset;

[0091] Fig.12 It is the confusion matrix of the test results of the present invention under the x-sub test benchmark on the NTU RGB+D dataset. DETAILED DESCRIPTION

[0092] The present invention will be further described below in conjunction with the accompanying drawings and specific implementation methods.

[0093] Example 1

[0094] like Figure 1 As shown, this embodiment provides an action recognition method based on the fusion of spatiotemporal graph convolutional network and Transformer network, including the following steps:

[0095] An action recognition method based on a fusion graph convolutional network and a Transformer network includes the following steps:

[0096] S1: preprocessing the acquired human skeleton sequence data, introducing joint velocity information based on the result obtained after the preprocessing, and obtaining a human skeleton sequence with the joint velocity information introduced, wherein the human skeleton sequence data is a frame sequence of the skeleton;

[0097] The specific steps are:

[0098] S1.1: Normalize the acquired human skeleton sequence based on the BN layer;

[0099] S1.2: Randomly crop the normalized human skeleton sequence to a length greater than 64 frames;

[0100] S1.3: using array sampling operations to perform up- and down-sampling operations on the spatial position array of the cropped human skeleton sequence in the time dimension, scaling the cropped human skeleton sequence to a length of 64 frames, thereby obtaining a scaled human skeleton sequence;

[0101] S1.4: Based on the scaled human skeleton sequence, the velocity information corresponding to each joint point is obtained by performing differential calculation on the previous and next frames. For the i-th joint point v in frame t, ti And the joint point v corresponding to this joint point in frame t+1 (t+1)i, and their spatial position information are respectively (x ti ,y ti , z ti ) and (x (t+1)i ,y (t+1)i , z (t+1)i ), then for the joint point v in frame t ti , its speed information is (x (t+1)i -x ti ,y (t+l)i -y ti , z (t+1)i -z ti ), the position information and speed information are spliced ​​together to obtain the preprocessed human skeleton sequence, and the spliced ​​joint point v ti The information can be expressed as: (x (t+1)i ,y (t+1)i , z (t+1)i , x (t+1)i -x ti ,y (t+1)i -y ti , z (t+1)i -z ti ), x, y, z represent three-dimensional coordinate points.

[0102] S2: Constructing a spatiotemporal skeleton graph based on the human skeleton sequence with joint velocity information;

[0103] The specific steps are:

[0104] S2.1: The preprocessed human skeleton sequence is represented by an undirected graph G = (V, E), wherein the human skeleton sequence has T frames and N joints, V in the undirected graph G represents the set of joints, and E represents the set of edges, i.e., the set of bone connections;

[0105] S2.2: V = {v ti |t=1,L,T,i=1,L,N} represents all the related nodes in the undirected graph G, t represents the frame index number, i represents the joint index number, T represents the total number of frames, and N represents the total number of joints;

[0106] S2.3: The set of edges E is given by the set of spatial edges E s and the set E of time edges t The two parts together constitute, that is, E=(E s , E t ), subset E s = {v ti v tj |(i, j)∈H} is the connection between joints in each frame space dimension, which belongs to the intra-frame connection, where H is the set of joint connections of the human body in space, and the subset E t = {v ti v(t+l)i} is the connection between the same joint point between the previous and next frames, which belongs to the inter-frame connection;

[0107] S2.4: Taking the point p in space as the center point, define the neighboring point set B(v ti )={v tj |d(v tj , v ti )≤D},d(v tj , v ti ) represents the jth joint point v in frame t tj To the i-th joint point v ti The shortest distance, D is a given threshold, which is 1. The neighboring point set of each joint point is marked to define the index order in the graph convolution process, that is, the joint point v ti The neighboring point set B(v ti ) is divided into three subsets according to the root node subset, the centripetal point subset, and the centrifugal point subset. The division rules and the defined index order are as follows:

[0108]

[0109] Among them, r i is the distance from joint point i to the center of gravity of the skeleton. Joint point i is the central joint point of the neighboring point set. The center of gravity of the skeleton is the coordinate position obtained by taking the average of the coordinate positions of all joint points of the entire human skeleton in a single frame. j is the distance from joint point j to the center of gravity of the skeleton;

[0110] After partitioning and defining the index order, a spatiotemporal skeleton graph is obtained.

[0111] S3: construct an adaptive spatiotemporal graph convolution unit module of the spatiotemporal skeleton graph and stack them to obtain a spatiotemporal graph convolution network for extracting local spatiotemporal features of the spatiotemporal skeleton graph;

[0112] The specific steps are:

[0113] S3.1: Based on the three subsets, a single-layer adaptive graph convolution layer is designed using an adaptive graph structure. The calculation formula is as follows:

[0114]

[0115] Among them, f in is the input of the spatial graph convolution layer, which refers to a single-layer adaptive graph convolution layer. out is the output of the spatial graph convolutional layer, W k is the weight matrix of the kth trainable subset. The number of partitions k of the root node subset, the centripetal point subset, and the centrifugal point subset is 3, that is, the normalized adjacency matrix of the spatiotemporal skeleton graph Divided into 3 partsAk , A1 is the unit matrix I, It is divided into two different directed graphs A2 and A3, A2 points to the center of gravity of the human skeleton, and A3 is far away from the center of gravity of the human skeleton. Among them, the degree matrix For the same value k, the adjacency matrix B k A k Initialize the adjacency matrix parameters and initialize the adjacency matrix B during the action recognition model training process. k The parameters are updated, and the adjacency matrix C k Relying on different human skeleton sequence sample data to generate graph structure information, the normalized Gaussian embedding function is used to capture the interaction between two joints to obtain the correlation dependency between the two joints. Since the normalized Gaussian embedding is equivalent to the softmax operation, the adjacency matrix C k The calculation formula is as follows:

[0116]

[0117] in, represents the inversion of the parameter matrix of the trainable linear layer θ of the kth subset, W φk represents the parameter matrix of the trainable linear layer φ of the kth subset, With W φk For use with f in Multiplication achieves data transformation.

[0118] S3.2: Constructing the residual channel attention layer The output features of the spatial graph convolution layer are fed into the introduced residual channel attention layer, that is, the features f output by the spatial graph convolution network are out Implementing the extrusion operation F sq (·), feature f out The number of channels is c-dimensional, and a c-dimensional vector S is obtained. The vector S is sent to two fully connected layers for excitation operation F. ex (·), get a new c-dimensional weight vector EE, and combine the vector EE with the spatial graph convolutional layer output feature f out After multiplication, it is output with the spatial graph convolution layer feature f out Perform residual connection and add to get the final output feature f; among them, the excitation operation F ex (·) Specifically, the C / r-dimensional vector is obtained through the first fully connected layer, and then ReLu activation is performed. Then the second fully connected layer is performed to convert the C / r-dimensional vector back to a C-dimensional vector, and Sigmoid activation is performed again to make the value between 0 and 1, and finally the c-dimensional weight vector EE is obtained.

[0119] S3.3: Use the traditional convolutional layer as the temporal convolutional layer to extract the local temporal features of the human skeleton sequence, and the convolution kernel size is 1×Γ, where Γ is the number of frames spanned by the temporal convolution kernel;

[0120] S3.4: The adaptive graph convolution layer, batch normalization layer, residual channel attention layer, ReLu activation function layer, temporal convolution layer, batch normalization layer and ReLu activation function layer are sequentially connected to form an adaptive spatiotemporal graph convolution unit module;

[0121] S3.5: Connect three adaptive spatiotemporal graph convolution unit modules in sequence, and set the temporal convolution step of each temporal convolution layer to 1, to obtain an adaptive spatiotemporal graph convolution unit module based on the stack, that is, the spatiotemporal graph convolution network.

[0122] S4: The spatiotemporal graph convolutional network and the visual Transformer network that extracts long-range temporal features from the pseudo image feature map and performs action classification are fused through a fusion layer, and the fused action recognition model is trained to obtain a trained action recognition model, wherein the fusion layer obtains the pseudo image feature map based on local spatiotemporal features;

[0123] The specific implementation steps of the fusion layer are:

[0124] S4.1: In the spatial dimension, that is, in the node dimension of the feature tensor, global average pooling is performed to aggregate the local spatiotemporal features in the spatial dimension of the joint points, so that the action features become time series features;

[0125] S4.2: The time series features are transformed into an array in chronological order from left to right and from top to bottom to obtain a two-dimensional representation of the time series features, which remains unchanged in the channel direction, and finally a pseudo image feature map is generated.

[0126] The specific steps of using the visual Transformer network to extract long-range temporal features from the pseudo image feature map and perform action classification are as follows: the pseudo image feature map is sent to the visual Transformer network for image classification, and the time series features are extracted through multiple encoder layers in the Transformer framework. After extraction, the output features are sent to the fully connected layer and connected to the softmax function to realize action category judgment.

[0127] Transformer framework can use many visual Transformer frameworks for image classification, such as Figure 8 ViT shown, such as Fig. 9The Swin Transformer shown in the figure. The Swin MLP framework is an adaptation of the Swin-Transformer network model. It uses an MLP layer (actually a group linear layer) to replace all MSA modules that perform attention calculations within the window to achieve Attention-free. Fig.10 As shown, the changed pseudo image feature map is sent to the Swin MLP framework to extract the temporal features. The hyperparameter settings for the number of channels C of the hidden layer in the first stage and the number of layers in each stage are: C = 96, layer numbers = {2, 2, 2}, and head numbers = {3, 6, 3}.

[0128] The stochastic gradient descent method with Nesterov momentum of 0.9 is used to train the action recognition model. The cross entropy is selected as the loss function for gradient calculation to realize back propagation, and the weight decay coefficient is set to l×e -4 .

[0129] S5: The human action skeleton sequence to be predicted is processed into a time skeleton graph and then sent to the trained action recognition model for prediction, and the action category is predicted. In order to support and verify the performance of the action recognition model proposed in the present invention, this embodiment is based on the pytorch deep learning framework to conduct simulation experiments on the NTU RGB+D public standard dataset. The other experimental parameters are set as follows: the total training round is epoch=150, the initial learning rate is 0.15, the learning rate is decayed to 0.2 times the previous one at the decay position (35, 55, 100), and the warmup learning rate is used in the first 5 epochs of model training. The Top-1 recognition accuracy is used as the evaluation index, and the model training and verification of the effectiveness of each module of the model are carried out under two test benchmarks: cross-performer (x-sub) and cross-view (x-view). The experimental results show that for the NTU RGB+D dataset, under the x-view test benchmark, the accuracy of the method disclosed by the present invention is 94.0%, and under the x-sub test benchmark, the accuracy of the method disclosed by the present invention is 87.4%. The confusion matrices of the test results under the x-view test benchmark and the x-sub test benchmark are as follows Fig.11 and Fig.12 shown.

[0130] The above are only representative embodiments of the present invention in many specific application scopes, and do not constitute any limitation on the protection scope of the present invention. Any technical solutions formed by transformation or equivalent replacement fall within the protection scope of the present invention.

Claims

1. An action recognition method based on a fusion graph convolutional network and a Transformer network, characterized in that: The steps include: S1: preprocessing the acquired human skeleton sequence data, introducing joint velocity information based on the result obtained after the preprocessing, and obtaining a human skeleton sequence with the joint velocity information introduced, wherein the human skeleton sequence data is a frame sequence of the skeleton; S2: Constructing a spatiotemporal skeleton graph based on the human skeleton sequence with joint velocity information; S3: construct an adaptive spatiotemporal graph convolution unit module of the spatiotemporal skeleton graph and stack them to obtain a spatiotemporal graph convolution network for extracting local spatiotemporal features of the spatiotemporal skeleton graph; S4: The spatiotemporal graph convolutional network and the visual Transformer network that extracts long-range temporal features from the pseudo image feature map and performs action classification are fused through a fusion layer, and the fused action recognition model is trained to obtain a trained action recognition model, wherein the fusion layer obtains the pseudo image feature map based on local spatiotemporal features; S5: Processing the human action skeleton sequence to be predicted into a temporal skeleton graph and then sending it to the trained action recognition model for prediction, and predicting the action category; The specific steps of step S1 are: S1.1: Normalize the acquired human skeleton sequence based on the BN layer; S1.2: Randomly crop the normalized human skeleton sequence to a length greater than 64 frames; S1.3: using array sampling operations to perform up- and down-sampling operations on the spatial position array of the cropped human skeleton sequence in the time dimension, scaling the cropped human skeleton sequence to a length of 64 frames, thereby obtaining a scaled human skeleton sequence; S1.4: Based on the scaled human skeleton sequence, the velocity information corresponding to each joint point is obtained by performing differential calculation on the previous and next frames. For the i-th joint point v in frame t, ti And the joint point v corresponding to this joint point in frame t+1 (t+1)i , and their spatial position information are respectively (x ti ,y ti ,z ti ) and (x (t+1)i ,y (t+1)i ,z (t+1)i ), then for the joint point v in frame t ti , its speed information is (x (t+1)i -x ti ,y (t+1)i -y ti ,z (t+1)i -z ti ), the position information and speed information are spliced ​​together to obtain the preprocessed human skeleton sequence, and the spliced ​​joint point v ti The information can be expressed as: (x (t+1)i ,y (t+1)i ,z (t+1)i ,x (t+1)i -x ti ,y (t+1)i -y ti ,z (t+1)i -z ti ), x, y, z represent three-dimensional coordinate points; The specific steps of step S2 are: S2.1: The preprocessed human skeleton sequence is represented by an undirected graph G = (V, E), where the human skeleton sequence has T frames and N joints. V in the undirected graph G represents the set of joints, and E represents the set of edges, i.e., the set of bone connections. S2.2: V = {v ti |t=1,…,T,i=1,…,N} represents all the related nodes in the undirected graph G, t represents the frame index number, i represents the joint point index number, T represents the total number of frames, and N represents the total number of joint points; S2.3: The set of edges E is given by the set of spatial edges E s and the set E of time edges t The two parts together constitute, that is, E=(E s ,E t ), subset E s = {v ti v tj |(i,j)∈H} is the connection between joints in each frame space dimension, which belongs to the intra-frame connection, where H is the set of joint connections of the human body in space, and the subset E t = {v ti v (t+1)i } is the connection between the same joint point between the previous and next frames, which belongs to the inter-frame connection; S2.4: Taking the point p in space as the center point, define the neighboring point set B(v ti )={v tj |d(v tj ,v ti )≤D},d(v tj ,v ti ) represents the jth joint point v in frame t tj To the i-th joint point v ti The shortest distance, D is a given threshold, which is 1. The neighboring point set of each joint point is marked to define the index order in the graph convolution process, that is, the joint point v ti The neighboring point set B(v ti ) is divided into three subsets according to the root node subset, the centripetal point subset, and the centrifugal point subset. The division rules and the defined index order are as follows: Among them, r i is the distance from joint point i to the center of gravity of the skeleton. Joint point i is the central joint point of the neighboring point set. The center of gravity of the skeleton is the coordinate position obtained by taking the average of the coordinate positions of all joint points of the entire human skeleton in a single frame. j is the distance from joint point j to the center of gravity of the skeleton; After partitioning and defining the index order, a spatiotemporal skeleton graph is obtained; The specific steps of step S3 are: S3.1: Based on the three subsets, a single-layer adaptive graph convolution layer is designed using an adaptive graph structure. The calculation formula is as follows: Among them, f in is the input of the spatial graph convolution layer, which refers to a single-layer adaptive graph convolution layer. out is the output of the spatial graph convolutional layer, W k is the weight matrix of the kth trainable subset. The number of partitions K of the root node subset, the centripetal point subset, and the centrifugal point subset is 3, which is the normalized adjacency matrix of the spatiotemporal skeleton graph. Divided into 3 partsA k , A1 is the unit matrix I, It is divided into two different directed graphs A2 and A3, A2 points to the center of gravity of the human skeleton, and A3 is far away from the center of gravity of the human skeleton. Among them, the degree matrix For the same value k, the adjacency matrix B k A k Initialize the adjacency matrix parameters and initialize the adjacency matrix B during the action recognition model training process. k The parameters are updated, and the adjacency matrix C k Relying on different human skeleton sequence sample data to generate graph structure information, the normalized Gaussian embedding function is used to capture the interaction between two joints to obtain the correlation dependency between the two joints. Since the normalized Gaussian embedding is equivalent to the softmax operation, the adjacency matrix C k The calculation formula is as follows: in, represents the inversion of the parameter matrix of the trainable linear layer θ of the kth subset, W φk represents the parameter matrix of the trainable linear layer φ of the kth subset, With W φk For use with f in Multiplication realizes data transformation; S3.2: Constructing the residual channel attention layer The output features of the spatial graph convolution layer are fed into the introduced residual channel attention layer, that is, the features f output by the spatial graph convolution network are out Implementing the extrusion operation F sq (·), feature f out The number of channels is c-dimensional, and a c-dimensional vector S is obtained. The vector S is sent to two fully connected layers for excitation operation F. ex (·), get a new c-dimensional weight vector EE, and combine the vector EE with the spatial graph convolutional layer output feature f out After multiplication, it is output with the spatial graph convolution layer feature f out Perform residual connection and add to get the final output feature f; S3.3: Use the traditional convolutional layer as the temporal convolutional layer to extract the local temporal features of the human skeleton sequence, and the convolution kernel size is 1×Γ, where Γ is the number of frames spanned by the temporal convolution kernel; S3.4: The adaptive graph convolution layer, batch normalization layer, residual channel attention layer, ReLu activation function layer, temporal convolution layer, batch normalization layer and ReLu activation function layer are sequentially connected to form an adaptive spatiotemporal graph convolution unit module; S3.5: Connect three adaptive spatiotemporal graph convolution unit modules in sequence, and set the temporal convolution step of each temporal convolution layer to 1, to obtain an adaptive spatiotemporal graph convolution unit module based on the stack, that is, the spatiotemporal graph convolution network.

2. According to claim 1, the action recognition method based on the fusion graph convolutional network and the Transformer network is characterized in that: The excitation operation F in step S3.2 ex (·) Specifically, the C / r-dimensional vector is obtained through the first fully connected layer, and then ReLu activation is performed. Then the second fully connected layer is performed to convert the C / r-dimensional vector back to a C-dimensional vector, and Sigmoid activation is performed again to make the value between 0 and 1, and finally the c-dimensional weight vector EE is obtained.

3. According to claim 2, the action recognition method based on the fusion graph convolutional network and the Transformer network is characterized in that: In step S4, the specific implementation steps of the fusion layer are: S4.1: In the spatial dimension, that is, in the node dimension of the feature tensor, global average pooling is performed to aggregate the local spatiotemporal features in the spatial dimension of the joint points, so that the action features become time series features; S4.2: The time series features are transformed into an array in chronological order from left to right and from top to bottom to obtain a two-dimensional representation of the time series features, which remains unchanged in the channel direction, and finally a pseudo image feature map is generated.

4. An action recognition system based on a fusion graph convolutional network and a Transformer network, characterized in that: include: Data processing module: preprocessing the acquired human skeleton sequence data, introducing joint velocity information based on the result obtained after preprocessing, and obtaining a human skeleton sequence with introduced joint velocity information, wherein the human skeleton sequence data is a frame sequence of the skeleton; Spatiotemporal skeleton graph construction module: constructs a spatiotemporal skeleton graph based on the human skeleton sequence with joint velocity information; Spatiotemporal graph convolutional network construction module: constructs the adaptive spatiotemporal graph convolution unit module of the spatiotemporal skeleton graph and stacks it. After stacking, a spatiotemporal graph convolutional network is obtained to extract the local spatiotemporal features of the spatiotemporal skeleton graph; Fusion module: The spatiotemporal graph convolutional network and the visual Transformer network that extracts long-range temporal features from the pseudo image feature map and performs action classification are fused through the fusion layer, and the action recognition model obtained after the fusion is trained to obtain a trained action recognition model. The fusion layer obtains the pseudo image feature map based on the local spatiotemporal features. Prediction module: The human action skeleton sequence to be predicted is processed into a time skeleton graph and then sent to the trained action recognition model for prediction to obtain the action category; The specific implementation steps of the data processing module are: S1.1: Normalize the acquired human skeleton sequence based on the BN layer; S1.2: Randomly crop the normalized human skeleton sequence to a length greater than 64 frames; S1.3: using array sampling operations to perform up- and down-sampling operations on the spatial position array of the cropped human skeleton sequence in the time dimension, scaling the cropped human skeleton sequence to a length of 64 frames, thereby obtaining a scaled human skeleton sequence; S1.4: Based on the scaled human skeleton sequence, the velocity information corresponding to each joint point is obtained by performing differential calculation on the previous and next frames. For the i-th joint point v in frame t, ti And the joint point v corresponding to this joint point in frame t+1 (t+1)i , and their spatial position information are respectively (x ti ,y ti ,z ti ) and (x (t+1)i ,y (t+1)i ,z (t+1)i ), then for the joint point v in frame t ti , its speed information is (x (t+1)i -x ti ,y (t+1)i -y ti ,z (t+1)i -z ti ), the position information and speed information are spliced ​​together to obtain the preprocessed human skeleton sequence, and the spliced ​​joint point v ti The information can be expressed as: (x (t+1)i ,y (t+1)i ,z (t+1)i ,x (t+1)i -x ti ,y (t+1)i -y ti ,z (t+1)i -z ti ), x, y, z represent three-dimensional coordinate points; The specific implementation steps of the spatiotemporal skeleton graph construction module are: S2.1: The preprocessed human skeleton sequence is represented by an undirected graph G = (V, E), where the human skeleton sequence has T frames and N joints. V in the undirected graph G represents the set of joints, and E represents the set of edges, i.e., the set of bone connections. S2.2: V = {v ti |t=1,…,T,i=1,…,N} represents all the related nodes in the undirected graph G, t represents the frame index number, i represents the joint point index number, T represents the total number of frames, and N represents the total number of joint points; S2.3: The set of edges E is given by the set of spatial edges E s and the set E of time edges t The two parts together constitute, that is, E=(E s ,E t ), subset E s = {v ti v tj |(i,j)∈H} is the connection between joints in each frame space dimension, which belongs to the intra-frame connection, where H is the set of joint connections of the human body in space, and the subset E t = {v ti v (t+1)i } is the connection between the same joint point between the previous and next frames, which belongs to the inter-frame connection; S2.4: Taking the point p in space as the center point, define the neighboring point set B(v ti )={v tj |d(v tj ,v ti )≤D},d(v tj ,v ti ) represents the jth joint point v in frame t tj To the i-th joint point v ti The shortest distance, D is a given threshold, which is 1. The neighboring point set of each joint point is marked to define the index order in the graph convolution process, that is, the joint point v ti The neighboring point set B(v ti ) is divided into three subsets according to the root node subset, the centripetal point subset, and the centrifugal point subset. The division rules and the defined index order are as follows: Among them, r i is the distance from joint point i to the center of gravity of the skeleton. Joint point i is the central joint point of the neighboring point set. The center of gravity of the skeleton is the coordinate position obtained by taking the average of the coordinate positions of all joint points of the entire human skeleton in a single frame. j is the distance from joint point j to the center of gravity of the skeleton; After partitioning and defining the index order, a spatiotemporal skeleton graph is obtained; The specific implementation steps of the spatiotemporal graph convolutional network construction module are: S3.1: Based on the three subsets, a single-layer adaptive graph convolution layer is designed using an adaptive graph structure. The calculation formula is as follows: Among them, f in is the input of the spatial graph convolution layer, which refers to a single-layer adaptive graph convolution layer. out is the output of the spatial graph convolutional layer, W k is the weight matrix of the kth trainable subset. The number of partitions K of the root node subset, the centripetal point subset, and the centrifugal point subset is 3, which is the normalized adjacency matrix of the spatiotemporal skeleton graph. Divided into 3 partsA k , A1 is the unit matrix I, It is divided into two different directed graphs A2 and A3, A2 points to the center of gravity of the human skeleton, and A3 is far away from the center of gravity of the human skeleton. Among them, the degree matrix For the same value k, the adjacency matrix B k A k Initialize the adjacency matrix parameters and initialize the adjacency matrix B during the action recognition model training process. k The parameters are updated, and the adjacency matrix C k Relying on different human skeleton sequence sample data to generate graph structure information, the normalized Gaussian embedding function is used to capture the interaction between two joints to obtain the correlation dependency between the two joints. Since the normalized Gaussian embedding is equivalent to the softmax operation, the adjacency matrix C k The calculation formula is as follows: in, represents the inversion of the parameter matrix of the trainable linear layer θ of the kth subset, W φk represents the parameter matrix of the trainable linear layer φ of the kth subset, With W φk For use with f in Multiplication realizes data transformation; S3.2: Constructing the residual channel attention layer The output features of the spatial graph convolution layer are fed into the introduced residual channel attention layer, that is, the features f output by the spatial graph convolution network are out Implementing the extrusion operation F sq (·), feature f out The number of channels is c-dimensional, and a c-dimensional vector S is obtained. The vector S is sent to two fully connected layers for excitation operation F. ex (·), get a new c-dimensional weight vector EE, and combine the vector EE with the spatial graph convolutional layer output feature f out After multiplication, it is output with the spatial graph convolution layer feature f out Perform residual connection and add to get the final output feature f; among them, the excitation operation F ex (·) Specifically, the C / r-dimensional vector is obtained through the first fully connected layer, and then ReLu activation is performed. Then, the second fully connected layer is performed to convert the C / r-dimensional vector back to a C-dimensional vector, and Sigmoid activation is performed again to make the value between 0 and 1, and finally the c-dimensional weight vector EE is obtained; S3.3: Use the traditional convolutional layer as the temporal convolutional layer to extract the local temporal features of the human skeleton sequence, and the convolution kernel size is 1×Γ, where Γ is the number of frames spanned by the temporal convolution kernel; S3.4: The adaptive graph convolution layer, batch normalization layer, residual channel attention layer, ReLu activation function layer, temporal convolution layer, batch normalization layer and ReLu activation function layer are sequentially connected to form an adaptive spatiotemporal graph convolution unit module; S3.5: Connect three adaptive spatiotemporal graph convolution unit modules in sequence, and set the temporal convolution step of each temporal convolution layer to 1, to obtain an adaptive spatiotemporal graph convolution unit module based on the stack, that is, the spatiotemporal graph convolution network.

5. The action recognition system based on the fusion graph convolutional network and Transformer network according to claim 4 is characterized in that: In the fusion module, the specific implementation steps of the fusion layer are: S4.1: In the spatial dimension, that is, in the node dimension of the feature tensor, global average pooling is performed to aggregate the local spatiotemporal features in the spatial dimension of the joint points, so that the action features become time series features; S4.2: The time series features are transformed into an array in chronological order from left to right and from top to bottom to obtain a two-dimensional representation of the time series features, which remains unchanged in the channel direction, and finally a pseudo image feature map is generated.

Images