Action recognition method based on graph convolutional neural network with attention mechanism

Abstract

The present invention discloses an action recognition method based on a graph convolutional neural network with an attention mechanism, comprising the following steps: step 1, obtaining video stream data of the type of human action to be recognized, obtaining human skeleton type data by a posture estimation algorithm as a human skeleton data set, step 2, constructing a coordination attention module, calculating the coordination features generated by the limbs and trunk during human movement, obtaining a center of gravity matrix with coordination features, and adding it to the human skeleton data set, step 3, inputting the human skeleton data set into a dual-stream graph convolutional neural network, and outputting predicted actions. And an importance attention module is added to the dual-stream graph convolutional neural network. The action recognition model of the present invention enables the final classification accuracy to be improved, making the existing dual-stream adaptive graph convolution model more suitable for action recognition tasks.

Description

Technical Field

[0001] The present invention relates to the technical field of video action recognition, and in particular to an action recognition method based on a graph convolutional neural network with an attention mechanism. Background Art

[0002] In the field of machine learning, action recognition is a very important task. It can be used in many scenarios in daily life, such as autonomous driving, human-computer interaction, public safety, etc. Therefore, this task has attracted more and more attention. At present, due to the explosive development of machine learning and deep learning in recent years, many high-performance action recognition algorithms have emerged, and the action recognition algorithm based on spatiotemporal graph convolution has achieved excellent results.

[0003] The theory of human movement balance describes that in order to prevent a person from falling during exercise, the body needs to constantly adjust its posture to keep the center of gravity basically unchanged. Athletes, in particular, can maintain their balance by swinging their arms and legs. For ordinary people, daily movements also require balance, and the coordination of the limbs and torso is required to prevent a person from falling to the ground. Therefore, when a person completes a certain movement, the limbs have a roughly fixed movement trajectory. For example, in the action of "running", when the left foot steps forward, the right arm must swing backward to keep the center of gravity unchanged, otherwise there will be a risk of falling.

[0004] In addition, different joints have different importance in different human movements, and there are often more than one important joint. Existing models fail to pay attention to the extraction of these features. In addition, due to the fixed physical connections of the human body, graph convolutional neural networks are often fixed when extracting features, and fail to pay better attention to the mutual features of several important joints from a global perspective. These joints are often not connected in most movements. For example, in the action of "clapping", from the human skeleton diagram, the nodes of the two hands are not directly connected and are far apart, but the two hands are an important part of the action of "clapping", and the changes in various features are also focused on the two hands.

[0005] Although skeleton-based action recognition algorithms have achieved excellent results on public datasets, current algorithms ignore the above two problems. Summary of the invention

[0006] In view of the deficiencies in the prior art, the present invention provides a method and device for action recognition based on a graph convolutional neural network with a dual attention mechanism. By combining the theory of human motion coordination, the coordination features generated during human motion are extracted. At the same time, the attention mechanism is used to extract more important joints from a global perspective, thereby effectively improving the recognition accuracy of the graph convolutional neural network in the field of action recognition.

[0007] To achieve the above object, the present invention adopts the following technical solutions:

[0008] The action recognition method of the graph convolutional neural network based on the attention mechanism of the present invention is characterized by comprising the following steps:

[0009] Step 1: Obtain video stream data of the human body motion type to be identified, and obtain human body skeleton type data using a posture estimation algorithm as a human body skeleton data set.

[0010] Step 2: construct a coordination attention module to calculate the coordination features of the limbs and trunk during human movement, obtain the center of gravity matrix with coordination features, and add it to the human skeleton dataset;

[0011] Step 3: The human skeleton dataset is input into the two-stream graph convolutional neural network and the predicted action is output.

[0012] Furthermore, the coordination attention module in step 2 is used to partition the human skeleton, calculate the center of gravity matrix and the coordination matrix, so as to obtain the center of gravity matrix with coordination features, which specifically includes the following steps:

[0013] Step 2.1, divide the human skeleton graph into five regions according to the structure of the human body. The five regions correspond to the head, left arm, right arm, left leg and right leg, respectively, and obtain five regional sub-graphs.

[0014] Step 2.2, calculate the centroid of each area;

[0015] Use the following formula to calculate the coordinates of the center of gravity of each area:

[0016]

[0017] In the formula, w x The horizontal coordinate of the center of gravity of the region, x n represents the horizontal coordinate of each joint point in the region, n represents the number of the joint point in the region, n = 1, 2, ..., n. In addition, the calculation method of the y-coordinate and z-coordinate of the center of gravity of the region is the same as the above formula.

[0018]

[0019]

[0020] The resulting centroid matrix is ​​shown in the following form:

[0021] (w1,w2,w3,w4,w5)

[0022] Step 2.3, calculate the coordination between two regions according to the covariance matrix.

[0023] The formula for calculating the covariance matrix is ​​as follows:

[0024] w i 、w j They represent (w1, w2, w3, w4, w5) respectively, i = 1, 2...5, as shown in the following form:

[0025] Where cov(·) represents the calculated result of covariance, w i and w j represents the coordinates of the center of gravity of the region, Represents the average coordinates of the centroid points of each region.

[0026] Step S2.4, calculate the coordination matrix according to the above formula, the form of the coordination matrix is ​​as follows:

[0027]

[0028] Step S2.5: According to step 2.4, three groups of 5×5 coordination matrices can be obtained. These three groups of coordination matrices are represented as w x ,w y , and w z , these three matrices can be used to represent the coordination characteristics of the body. x ,w y , and w z Compress to the same size as the centroid matrix. The compression method here is to add columns one by one. Taking the first column as an example, the compression method is shown in the following form:

[0029] w1′=cov(w1,w1)+cov(w2,w1)+cov(w3,w1)+cov(w4,w1)+cov(w5,w1)

[0030] Step S2.6, add the centroid matrix to the compressed compatibility matrix to obtain the centroid matrix with compatibility characteristics

[0031] in,

[0032]

[0033]

[0034]

[0035]

[0036] Furthermore, due to the limitations of the human body topology, it is difficult for the graph convolution model to learn the relationship between various terminal nodes, which are often important components of the action. In addition, deep graph convolution models are prone to excessive smoothing of features, so deep models are not suitable. The present invention introduces an attention mechanism into the model to obtain the importance features of the nodes in the feature graph and transfer them to the original feature graph. The present invention proposes an importance attention module. When extracting features by importance, this module does not use the adjacency matrix, but operates directly on the feature graph, which can effectively overcome the limitations of graph convolutional neural networks.

[0037] The dual-stream graph convolutional neural network in step 3 includes an importance attention module, and a dual-stream graph convolutional neural network with an importance attention module is constructed; including the following steps

[0038] Step 3.1, construct an adaptive graph convolution module, which includes a sequentially connected spatial graph convolution layer convs, a temporal graph convolution layer convert, an additional random dropout process Dropout and a residual connection; in addition, the spatial graph convolution layer and the temporal convolution layer are each connected to a batch normalization layer and an activation function layer;

[0039] Step 3.2, build an adaptive graph convolution module with an importance attention module, including a sequentially connected spatial graph convolution layer convs, an importance attention module, a temporal graph convolution layer convt, an importance attention module, an additional random dropout process Dropout and a residual connection; in addition, the two importance attention modules are each connected to a batch normalization layer and an activation function layer;

[0040] Step 3.3: Build J-stream module and B-stream module

[0041] The J-stream module has the same structure as the B-stream module, which includes a data BN layer, 9 adaptive graph convolution modules and 1 adaptive graph convolution module with importance attention module. The data BN layer is added before the 10 adaptive graph convolution modules to standardize the input data.

[0042] Step 3.4: Build a two-stream graph convolutional neural network with importance attention module

[0043] The dual-stream graph convolutional neural network with an importance attention module includes a J-stream module, a B-stream module, an importance attention module and two SoftMax layers; a human skeleton dataset with coordination features is input into the dual-stream graph convolutional neural network, the human skeleton dataset is first processed to generate joint data and bone data, which are respectively input into the J-stream module and the B-stream module, and then a global average pooling layer is executed to pool feature maps of different samples to the same size, and then they pass through the importance attention modules respectively to reach the two SoftMax layers to obtain the classification scores of the two streams, and then the two scores are added to obtain a fused score and predict the action label.

[0044] Furthermore, the importance attention module in step 3 includes two convolutional layers and one Softmax layer. The specific data processing process is as follows:

[0045] Step 3.1.1, send feature map A into two convolutional layers to obtain two new feature maps B and C. Where N×M is the product of the batch size and the number of characters, C represents the number of channels, T represents the number of action frames, and V represents the number of joint points.

[0046] Step 3.1.2, reorganize the feature maps B and C into Where D = (N × M) × T × V, which represents the number of feature points on each channel.

[0047] Step 3.1.3, perform matrix multiplication on the transpose of feature map B and C and use the Softmax layer to calculate the position attention feature map S, where

[0048] The calculation formula of the attention feature map is as follows:

[0049]

[0050] s ji Represents the influence of the i-th position on the feature map on the feature of the j-th position.

[0051] Step 3.1.4, reorganize the feature map S and multiply it by the scale factor α, add it to the input feature map A, and get the final output feature M, where The initial value of α is set to 0, and larger weights can be gradually learned.

[0052] The specific calculation method is expressed as follows:

[0053]

[0054] E jIt means that the feature M of each position is the weighted sum of all position features and the original features. Therefore, it has a global view and can selectively aggregate contextual information based on the spatial attention map.

[0055] Furthermore, the spatial graph convolution layer convs described in step 3.1 is used to transform the topological structure of the network from the adjacency matrix A to the convolutional layer convs in an end-to-end learning manner. k Decide on the following form:

[0056]

[0057] K in the formula v Set to 3, W k is the weight matrix, A k is an N×N adjacency matrix that represents the physical structure of the human body. k It is also an N×N adjacency matrix, C k Represents a data correlation graph, which is used to learn a unique graph for each sample. In order to determine whether there is a connection between two adjacent nodes and how strong the connection is, the model uses a normalized Gaussian function to calculate the similarity between the two nodes, as shown in the following form:

[0058]

[0059] N in the formula represents the number of all nodes. The model uses dot product to calculate the similarity between two nodes in the embedding space, and the obtained C k It is an N×N similarity matrix, and the model normalizes the values ​​to between 0 and 1.

[0060] In the second aspect, an embodiment of the present invention proposes a graph convolutional neural network structure based on an attention mechanism, including a human skeleton dataset generation module, a coordinated attention module and a dual-stream graph convolutional neural network, a dual-stream graph convolutional neural network.

[0061] The human skeleton data set generation module is used to obtain the video stream data of the human action type to be identified, use the existing posture estimation algorithm to process the imported video stream data, obtain the human skeleton type data and human skeleton graphics, and generate the coordinates and confidence features of each key node to generate the human skeleton data set;

[0062] The coordination feature extraction module is used to calculate the coordination features generated by human body movement. The human skeleton map is divided into five partitions. The definition of the center of gravity is combined with the definition of the covariance matrix. The relationship between the centers of gravity on each partition is calculated to obtain a set of coordination feature matrices, which are then assigned to the original feature map, so that the input data of the model has the coordination features of the human body.

[0063] The two-stream graph convolutional neural network is used to perform action recognition on a human skeleton dataset containing coordination features. The two-stream graph convolutional neural network also includes an important feature extraction module, which uses an improved spatial attention mechanism to extract the more critical joints of human motion within a global perspective. This module operates directly on the feature map, which can overcome the limitations of topological map connections and extract joints that are more important to the composition of the action from a global perspective.

[0064] Beneficial effects of the present invention: On the one hand, the present invention is inspired by the theory of human motion coordination, and designs a module that can extract the coordination features generated during human motion. The module calculates the center of gravity of each partition of the human body by using the definition of the center of gravity in mathematics and physics, and uses the covariance matrix to calculate the relationship between the two of them. This relationship can be used to represent the coordination features generated during motion. On the other hand, in view of the drawbacks of the existing graph convolution model, combined with the existing attention mechanism, an improved spatial attention module is proposed, which is used to learn more important joint features on a global scale. The action recognition model of the present invention improves the final classification accuracy, making the existing two-stream adaptive graph convolution model more suitable for action recognition tasks. BRIEF DESCRIPTION OF THE DRAWINGS

[0065] Figure 1 A human skeleton partitioning strategy diagram according to an embodiment of the present invention.

[0066] Figure 2 Schematic diagram of a coordinated attention module according to an embodiment of the present invention.

[0067] Figure 3 Schematic diagram of an adaptive graph convolution model according to an embodiment of the present invention.

[0068] Figure 4 Schematic diagram of an action recognition model of a graph convolutional neural network based on a dual attention mechanism according to an embodiment of the present invention.

[0069] Figure 5 Schematic diagram of an importance attention module according to an embodiment of the present invention.

[0070] Figure 6 Schematic diagram of an adaptive graph convolution model integrating an importance attention module according to an embodiment of the present invention. DETAILED DESCRIPTION

[0071] The present invention will now be described in further detail with reference to the accompanying drawings.

[0072] The action recognition method based on graph convolutional neural network with attention mechanism includes the following steps:

[0073] Step 1: obtain the video stream data of the human body action type to be identified, use the existing posture estimation algorithm to process the imported video stream data, and generate a human skeleton data set.

[0074] Step 2: construct a coordination attention module to calculate the coordination features generated by the limbs and trunk during human movement, and add the coordination features to the human skeleton dataset;

[0075] The coordination attention module is a calculation unit used to partition the human skeleton, calculate the center of gravity matrix and the coordination matrix, so as to obtain the center of gravity matrix with coordination features. Specifically, it includes the following steps:

[0076] Step 2.1, such as Figure 1 As shown, the human skeleton graph is divided into five regions according to the structure of the human body. The five regions correspond to the head, left arm, right arm, left leg and right leg, respectively, and five regional sub-graphs are obtained.

[0077] Step 2.2, according to the definition of the center of gravity in mathematics and physics, calculate the center of gravity of each area. In order to reduce the amount of calculation of the model, this module uses the center of gravity of each area to calculate the coordination. This will be much smaller than directly using the joint calculation, and it can also effectively avoid the problem of inconsistent nodes in each part. Use the following formula to calculate the coordinates of the center of gravity of each area:

[0078]

[0079] Where, the horizontal coordinate of the center of gravity of the w region, x n represents the horizontal coordinate of each joint point in the region, n represents the number of the joint point in the region, n = 1, 2, ..., n. In addition, the calculation methods of the y coordinate and the z coordinate are the same as the above formula.

[0080]

[0081]

[0082] The final centroid matrix is ​​shown in the following form:

[0083] (w1,w2,w3,w4,w5)

[0084] Step S2.3, according to the definition of the covariance matrix, it can be used to calculate the coordination between two regions. The present invention rewrites the covariance matrix calculation formula, let X i , X j They are respectively equal to (w1, w2, w3, w4, w5), i = 1, 2...5, as shown in the following form:

[0085]

[0086] Where cov(·) represents the calculated result of covariance, w i and w j represents the coordinates of the center of gravity of the region, It represents the average coordinates of the centroid points of each region. The values ​​of i and j can be equal.

[0087] Step S2.4, calculate the coordination matrix according to the above formula, the form of the coordination matrix is ​​as follows:

[0088]

[0089] Step S2.5: According to step 2.4, three groups of 5×5 coordination matrices can be obtained. These three groups of coordination matrices are represented as w x ,w y , and w z , these three matrices can be used to represent the coordination characteristics of the body. x ,w y , and w z Compress to the same size as the centroid matrix. The compression method here is to add columns one by one. Taking the first column as an example, the compression method is shown in the following form:

[0090]

[0091] Step S2.6, add the centroid matrix to the compressed compatibility matrix to obtain the centroid matrix with compatibility characteristics

[0092] in,

[0093]

[0094]

[0095]

[0096]

[0097] Step 3: Construct an adaptive graph convolution layer with importance attention mechanism. Figure 3 FIG. 1 is a schematic diagram of an adaptive graph convolution model according to an embodiment of the present invention. Figure 4 Schematic diagram of an action recognition model of a graph convolutional neural network based on a dual attention mechanism according to an embodiment of the present invention. Figure 5 This is a schematic diagram of an importance attention module according to an embodiment of the present invention. Figure 6 It is an adaptive graph convolution model with an importance attention module of the present invention.

[0098] Step 3.1, building an adaptive graph convolution model with an importance attention module, including a sequentially connected spatial graph convolution layer convs, a first importance attention module, a temporal graph convolution layer convt, a second importance attention module, an additional random dropout process Dropout and a residual connection; wherein Dropout is set to 0.5; the first importance attention module and the second importance attention module are each connected with a batch normalization layer and an activation function layer;

[0099] Step 3.1.1, the feature map A obtained after the spatial graph convolution, Where N×M is the product of the batch size and the number of characters, C represents the number of channels, T represents the number of action frames, and V represents the number of joints. Then, the feature map A is sent to two 1×1 convolutional layers to obtain two new feature maps B and C.

[0100] Step 3.1.2, reorganize the feature maps B and C into Here D = (N×M)×T×V, which represents the number of feature points on each channel.

[0101] Step 3.1.3, perform matrix multiplication on the transpose of B and C and use the Softmax layer to calculate the position attention feature map S, where The calculation formula of the attention feature map is as follows:

[0102]

[0103] s ji Represents the influence of the i-th position on the feature map on the feature of the j-th position.

[0104] Step 3.1.4, reorganize the feature map S and multiply it by the scale factor α, add it to the feature map A to get the final output feature M, where The initial value of α is set to 0, and it can gradually learn larger weights. Its representation is as follows:

[0105]

[0106] E j It means that the feature M of each position is the weighted sum of all position features and the original features. Therefore, it has a global view and can selectively aggregate contextual information based on the spatial attention map.

[0107] S4, construct a two-stream graph convolutional neural network with an importance attention module, use the data set obtained in step 1 as the input data of the two-stream graph convolutional neural network, and predict the action label as the output data. Integrate the two attention modules in steps S2 and S3 into the two-stream graph convolutional neural network, such as Figure 4 shown.

[0108] Step S4.1, build the spatial graph convolution layer convs; used to transform the network topology from the adjacency matrix A to the convolutional layer convs in an end-to-end learning manner. k Decide on the following form:

[0109]

[0110] K in the formula v Set to 3, W k is the weight matrix, A k is an N×N adjacency matrix that represents the physical structure of the human body. k is also an N×N adjacency matrix, but B k There are no specific constraints on the values, which means that the graph is completely learned from the training data. In data-driven tasks, the model can completely learn the graph from the target task. Considering B k The value in can be any value, which can not only represent the physical structure of the human body, but also the connection strength between adjacent nodes. k It can represent a data correlation graph, which can learn a unique graph for each sample. In order to determine whether there is a connection between two adjacent nodes and how strong the connection is, the model uses a normalized Gaussian function to calculate the similarity of the two nodes, as shown in the following form:

[0111]

[0112] N in the formula represents the number of all nodes. The model uses dot product to calculate the similarity between two nodes in the embedding space, and the obtained C k It is an N×N similarity matrix, and the model normalizes the values ​​to between 0 and 1.

[0113] Step S4.2, building an adaptive graph convolution module; the adaptive graph convolution module includes a sequentially connected spatial graph convolution layer convs, a temporal graph convolution layer convert, an additional random dropout process Dropout and a residual connection; wherein the Dropout is set to 0.5; a batch normalization layer and an activation function layer are each connected after the spatial graph convolution layer and the temporal convolution layer;

[0114] Step S4.3, build an adaptive graph convolution module with an importance attention module; based on the original adaptive graph convolution module, this module adds the importance attention module proposed in step S3 after the spatial graph convolution and the temporal convolution respectively;

[0115] Step S4.4, by stacking the adaptive graph convolution modules, an adaptive graph convolution network is constructed. The model includes 10 layers in total, including 9 adaptive graph convolution modules and 1 adaptive graph convolution module with importance attention module proposed in step 4.3. The output channels of each module are 64, 64, 64, 128, 128, 128, 256, 256, 256 and 256 respectively; a data BN layer is added at the beginning to standardize the input data, and a global average pooling layer is performed after the end of the 10th layer to pool the feature maps of different samples to the same size. The final model output is sent to the SoftMax classifier to obtain the prediction result;

[0116] Step S4.5, build an action recognition model of a graph convolutional neural network with a dual attention mechanism; divide the skeleton data with coordination features into joint stream data and skeleton stream data and input them into the model built in step S4.4 respectively, obtain the SoftMax classification scores of the two streams respectively, and then add the two scores to obtain the fused score and predict the action label.

[0117] The present invention also proposes a graph convolutional neural network based on the attention mechanism, such as Figure 4 As shown, the network includes a human skeleton dataset generation module, a coordinated attention module and a dual-stream graph convolutional neural network.

[0118] The human skeleton data set generation module is used to obtain the video stream data of the human action type to be identified, use the existing posture estimation algorithm to process the imported video stream data, obtain the human skeleton type data and human skeleton graphics, and generate the coordinates and confidence features of each key node to generate the human skeleton data set;

[0119] The coordination attention module is used to calculate the coordination features generated by human body movement. The human skeleton graph is divided into five partitions. The definition of the center of gravity is combined with the definition of the covariance matrix. The relationship between the centers of gravity on each partition is calculated to obtain a set of coordination feature matrices, which are then assigned to the original feature graph, so that the input data of the model has the coordination features of the human body.

[0120] The two-stream graph convolutional neural network building module is used to build a two-stream graph convolutional neural network, in which joint data and bone data are used as input features of the two streams respectively, and the predicted action label is used as the output data.

[0121] The dual-stream graph convolutional neural network includes an importance attention module, which uses an improved spatial attention mechanism to extract the more critical joints of human body movement within a global perspective. This module operates directly on the feature map, which can overcome the limitations of topological map connections and extract joints that are more important to the action composition from a global perspective.

[0122] Through the network of the second embodiment of the present invention, the transmission object is determined by resuming the data inclusion relationship of the entire application, so as to achieve the goal of identifying human body movements in the video stream. The network provided by the embodiment of the present invention can execute the action recognition method based on the adaptive graph convolutional neural network provided by any embodiment of the present invention, and has the corresponding functions and beneficial effects of the execution method.

Claims

1. The action recognition method based on graph convolutional neural network with attention mechanism is characterized by: The steps include: Step 1, obtaining video stream data of the human body motion type to be identified, and obtaining human body skeleton type data by a posture estimation algorithm as a human body skeleton data set; Step 2: construct a coordination attention module to calculate the coordination features of the limbs and trunk during human motion, obtain the center of gravity matrix with coordination features, and add it to the human skeleton dataset. The specific steps include: Step 2.1, divide the human skeleton graph into five regions according to the structure of the human body. The five regions correspond to the head, left arm, right arm, left leg and right leg, respectively, and obtain five regional sub-graphs; Step 2.2, calculate the centroid of each area; Use the following formula to calculate the coordinates of the center of gravity of each area: In the formula, w x The horizontal coordinate of the center of gravity of the region, x n represents the horizontal coordinate of each joint point in the region, n represents the number of the joint point in the region, n = 1, 2, ..., n; in addition, the calculation method of the y-coordinate and z-coordinate of the center of gravity of the region is the same as the above formula; The resulting centroid matrix is ​​shown in the following form: (w1,w2,w3,w4,w5) Step 2.3, calculate the coordination between two regions according to the covariance matrix; The formula for calculating the covariance matrix is ​​as follows: w i 、w j They represent (w1, w2, w3, w4, w5) respectively, i = 1, 2...5, as shown in the following form: Where cov(·) represents the calculated result of covariance, w i and w j represents the coordinates of the center of gravity of the region, It represents the average coordinates of the centroid points of each region; Step S2.4, calculate the coordination matrix according to the above formula, the form of the coordination matrix is ​​as follows: Step S2.5: According to step 2.4, three groups of 5×5 coordination matrices can be obtained. These three groups of coordination matrices are represented as w x ,w y , and w z , these three matrices can be used to represent the coordination characteristics of the body; x ,w y , and w z Compress to the same size as the centroid matrix. The compression method here is to add columns one by one. Taking the first column as an example, the compression method is shown in the following form: w1′=cov(w1,w1)+cov(w2,w1)+cov(w3,w1)+cov(w4,w1)+cov(w5,w1) Step S2.6, add the centroid matrix to the compressed compatibility matrix to obtain the centroid matrix with compatibility characteristics in, Step 3: The human skeleton dataset is input into the two-stream graph convolutional neural network and the predicted action is output.

2. The action recognition method based on graph convolutional neural network with attention mechanism according to claim 1 is characterized in that: The dual-stream graph convolutional neural network in step 3 includes an importance attention module, and a dual-stream graph convolutional neural network with an importance attention module is constructed; including the following steps Step 3.1, construct an adaptive graph convolution module, which includes a sequentially connected spatial graph convolution layer convs, a temporal graph convolution layer convert, an additional random dropout process Dropout and a residual connection; in addition, the spatial graph convolution layer and the temporal convolution layer are each connected to a batch normalization layer and an activation function layer; Step 3.2, build an adaptive graph convolution module with an importance attention module, including a sequentially connected spatial graph convolution layer convs, an importance attention module, a temporal graph convolution layer convt, an importance attention module, an additional random dropout process Dropout and a residual connection; in addition, the two importance attention modules are each connected to a batch normalization layer and an activation function layer; Step 3.3: Build J-stream module and B-stream module The J-stream module has the same structure as the B-stream module, which includes a data BN layer, 9 adaptive graph convolution modules, and 1 adaptive graph convolution module with importance attention module. The data BN layer is added before the 10 adaptive graph convolution modules to standardize the input data. Step 3.4: Build a two-stream graph convolutional neural network with importance attention module The two-stream graph convolutional neural network with importance attention module includes J-stream module, B-stream module, importance attention module and two SoftMax layers; The human skeleton dataset with coordination features is input into the dual-stream graph convolutional neural network. The human skeleton dataset is first processed to generate joint data and skeleton data, which are respectively input into the J stream module and the B stream module. After completion, the global average pooling layer is executed to pool the feature maps of different samples to the same size. After passing through the importance attention modules, they reach the two SoftMax layers to obtain the classification scores of the two streams. The two scores are then added together to obtain the fused score and predict the action label.

3. The action recognition method based on graph convolutional neural network with attention mechanism according to claim 2 is characterized in that: The importance attention module consists of two convolutional layers and one Softmax layer. The specific data processing process is as follows: Step 3.1.1, send feature map A into two convolutional layers to obtain two new feature maps B and C. Where N×M is the product of the batch size and the number of characters, C represents the number of channels, T represents the number of action frames, and V represents the number of joint points; Step 3.1.2, reorganize the feature maps B and C into Where D = (N × M) × T × V, which represents the number of feature points on each channel; Step 3.1.3, perform matrix multiplication on the transpose of feature map B and C and use the Softmax layer to calculate the position attention feature map S, where The calculation formula of the attention feature map is as follows: s ji Represents the influence of the i-th position on the feature map on the j-th position feature; Step 3.1.4, reorganize the feature map S and multiply it by the scale factor α, add it to the input feature map A, and get the final output feature M, where The initial value of α is set to 0, and it can gradually learn larger weights; The specific calculation method is expressed as follows: E j It means that the feature M of each position is the weighted sum of all position features and the original features.

4. The action recognition method based on graph convolutional neural network with attention mechanism according to claim 2 is characterized in that: The spatial graph convolution layer convs is used to transform the topological structure of the network from the adjacency matrix A in an end-to-end learning manner. k Decide on the following form: K in the formula v Set to 3, W k is the weight matrix, A k is an N×N adjacency matrix, which represents the physical structure of the human body; B k It is also an N×N adjacency matrix, C k Represents a data correlation graph, which is used to learn a unique graph for each sample. In order to determine whether there is a connection between two adjacent nodes and how strong the connection is, the model uses a normalized Gaussian function to calculate the similarity between the two nodes, as shown in the following form: N in the formula represents the number of all nodes; the model uses dot product to calculate the similarity between two nodes in the embedding space, and the obtained C k It is an N×N similarity matrix, and the model normalizes the values ​​to between 0 and 1.

Images