Human behavior recognition method based on GCN and transformer

Abstract

The application discloses a human behavior recognition method based on GCN and a transformer, and specifically relates to the following steps: acquiring an RGB video containing human motion, extracting key frames and performing pretreatment, cropping the region of human activity, labeling the video so that it has corresponding category labels; processing each frame of image, recognizing the key points of the human body to form a skeleton sequence, and supplementing the sequence to divide a training set and a test set; inputting the training set into a GCN and a transformer network for training; inputting the test set into the GCN and the transformer network for testing, so that human behavior recognition can be realized. The application utilizes the global modeling capability of the transformer, extracts features in the spatial dimension and the time dimension, and further performs spatio-temporal fusion on the extracted features in the global dimension, so that the model can better understand the feature information and the recognition rate is improved.

Description

Technical Field

[0001] This invention belongs to the field of human behavior recognition technology, specifically relating to a human behavior recognition method based on GCN and Transformer. Background Technology

[0002] As a hot topic in computer vision research, human behavior recognition has been studied extensively for decades and has made significant progress. This technology not only has important academic significance but also broad application prospects, with applications in fields such as security monitoring systems, medical diagnosis and monitoring, and intelligent human-computer interaction.

[0003] Skeleton-based algorithms are the mainstream algorithms for human behavior recognition. They identify individual behavior patterns by analyzing the dynamic changes of the human skeleton. They have the characteristics of small data volume, sufficient information expression, strong robustness, and resistance to interference. In particular, the development of pose estimation algorithms (such as OpenPose) and human skeleton capture system sensors (such as Kinect) in recent years has greatly promoted the development of skeleton-based behavior recognition algorithms. Human skeleton data is a graph topology with a natural structure. This structural characteristic enables graph convolution algorithms (GCN) to process data well, extract the potential relationships between various skeletal joints when the human body makes a behavior, and model and learn the relationships between skeleton nodes, thereby representing the structural information and motion characteristics of the human skeleton. Therefore, graph convolution algorithms have been widely used in skeleton-based behavior recognition. In addition, existing algorithms often use standard convolution methods to extract information in the time dimension. Their shortcomings are: (1) they cannot determine an appropriate size of convolution kernel so that the algorithm can learn the temporal features of the sequence more effectively; (2) they only extract short-term temporal information and cannot model the relationships between long-distance video frames. Summary of the Invention

[0004] The purpose of this invention is to propose a human behavior recognition method based on GCN and Transformer, which solves the problem of low recognition rate caused by the inability to globally model information in the time dimension in the existing technology.

[0005] The technical solution adopted in this invention is a human behavior recognition method based on GCN and Transformer, which is implemented according to the following steps:

[0006] Step 1: Acquire multiple RGB videos containing human movement, extract multiple keyframes from each RGB video, preprocess the extracted keyframe images, crop out the areas containing only human activity, and then label the videos so that each behavior video has a corresponding category label.

[0007] Step 2: Use the human pose estimation algorithm MMPose to process each frame of the image, identify the key points of the human body in the image, arrange the skeleton data in chronological order to form a skeleton sequence, and fill in the sequences of different lengths. Finally, divide the training set and the test set.

[0008] Step 3: Input the training set into the GCN and Transformer networks for training;

[0009] Step 4: Input the test set into the GCN and Transformer networks for testing to achieve human behavior recognition.

[0010] The invention is further characterized in that,

[0011] In step 1, the formula for keyframe extraction is shown in equation (1);

[0012] N = T / f (1);

[0013] Where N is the total number of video frames; T is the video duration in seconds; and f is the frame rate.

[0014] Step 2 specifically involves: using the human pose estimation algorithm MMPose to process each frame of the image, identifying key points of the human body in the image, namely the positions of joints and the corresponding connecting skeletons, and then using three-dimensional coordinates (x, y, z) to represent the position of each key point. i ,y i ,z i The sequence is represented as ), where i = 1, 2, 3…25; i is the number of keypoints. The skeleton data is then arranged in chronological order to form a skeleton sequence. Numerical padding is used to standardize the sequence length, ensuring that the length of all sequences reaches the longest possible length, L. max Then, 70% of the skeleton sequences were used as the training set and 30% of the skeleton sequences were used as the test set.

[0015] Step 3 specifically involves:

[0016] Step 3.1: Use the GCN method to extract spatial features from the skeleton sequence corresponding to each keyframe;

[0017] Step 3.2: Use Transformer to extract the temporal information of the sequence.

[0018] Step 3.3: Perform multi-head self-attention operation on the obtained vectors Q, K, and V;

[0019] Step 3.4: After extracting features using the GCN and Transformer methods, the resulting vector Z is... oitThe dimensions are transformed to combine the spatiotemporal features, and the resulting features are modeled globally using Transformer;

[0020] Step 3.5: Classify each vector into the category label corresponding to the one in Step 1 using the nn.Conv2d() method; each vector will have num_class numbers, which are converted into probabilities between 0 and 1 using the softmax() function; among the probabilities obtained by the network, the probability that the vector actually belongs to the category is p. * Let the video sample be x. i (i = 1, 2, 3…n), where n is the number of samples, according to probability p… * Construct x i The focal loss function is constructed, and after constructing the loss function, the network hyperparameters are optimized through iterative training using the Adam optimizer and backpropagation algorithm.

[0021] Step 3.1 specifically involves:

[0022] In an RGB video, the relative coordinates D between various key points within the same frame are... R The relative coordinates D between the same key point in different frames of time. T The vector Z is integrated into the corresponding skeleton sequence in step 2 to form a new representation vector Z. The vector Z is then divided into four parts according to the human skeletal structure: upper limbs, lower limbs, upper trunk and lower trunk. Then, the combination of each joint point of each part is convolved, as shown in equation (2).

[0023]

[0024] Among them, X i (i = 1, 2, 3, 4) represents the combined data for each joint; v tj A represents each joint in each combination. i Let D represent the adjacency matrix for each combination. i w represents the degree matrix of each combination. ij This represents the weight of each key point in the combination; after extracting information, a vector X representing the features of each combination is obtained. i Then consider the four vectors X i We perform weighted summation to obtain a higher-level feature vector Z, as shown in equation (3);

[0025]

[0026] Step 3.2 specifically involves:

[0027] Step 3.21: After GCN extracts information, it forms a vector Z. The dimension of vector Z is transformed to (B*V,T,C), where B represents the batch size, T represents the sequence length, C represents the number of channels, and V represents the number of joints in the skeleton diagram. Cosine encoding is performed on each joint of each frame image in the skeleton sequence to obtain the order vector PE representing each frame image, as shown in equations (4) and (5).

[0028]

[0029]

[0030] Where d represents the dimension of the vector, i.e., C, PE (pos,2i) PE represents the even-numbered dimension of a vector. (pos,2i+1) The odd-numbered dimension of the vector is represented, and each pos represents the absolute position of each frame in the sequence.

[0031] Step 3.22: Add the sequence vector PE to the vector Z to obtain a new representation vector E. From this point on, the vector of each frame contains not only feature information but also sequence information, as shown in equation (6):

[0032] E = Z + PE (6);

[0033] Step 3.23: Perform linear transformations on the representation vector E to obtain the query vector Q, the key value vector K, and the representative value vector V, as shown in equations (7)-(9):

[0034] Q = W Q ·LN(Z) (7);

[0035] K = W K ·LN(Z) (8);

[0036] V = W V ·LN(Z) (9);

[0037] Where LN represents LayerNorm, W Q W K W V Let Q = (q1, q2, ..., q3) represent the weight matrices used to generate three different vectors. D ), K = (k1, k2…k D V = (v1, v2, ... v) D ); D represents the number of heads in the multi-head self-attention mechanism.

[0038] Step 3.3 specifically involves:

[0039] Step 3.31, calculate the attention Atten, as shown in Equation (10):

[0040]

[0041] Here, Q is a set of vectors used to find relevant information. The query vector can be viewed as a process of asking questions about each element in the input sequence to determine its correlation with other elements. The key vector K contains the input information at each position, representing each element in the sequence answering the question to the query vector Q. d represents the dimension of the query vector Q and the key vector K, and SM represents softmax().

[0042] Step 3.32: Embed the attention Atten into the value vector V; finally, add it to the vector Z to obtain the vector Z1, as shown in equation (11):

[0043] Z1=Z+Atten·V (11);

[0044] Step 3.33: The vector Z1 is processed through a two-layer feedforward neural network structure. Specifically, in the first layer of the feedforward network structure, Z1 is linearly transformed by multiplying the vector Z1 by the weight matrix W1 and adding the bias vector b1 to obtain the vector Z2, as shown in equation (12):

[0045] Z2 = W1Z1 + b1 (12);

[0046] Then, the vector Z2 is activated by the ReLU function to capture its nonlinear characteristics, resulting in the vector Z3, as shown in equation (13):

[0047] Z3 = max(0, Z2) (13);

[0048] Finally, the vector Z3 is passed through the second layer of the feedforward neural network; first, the vector Z3 is multiplied by the weight matrix W2, and then the bias vector b2 is added to obtain the vector Z. out As shown in equation (14):

[0049] Z out =W2Z3+b2 (14).

[0050] Step 3.4 specifically involves:

[0051] Vector Z out The dimensions are then reconstructed to (B, V*T, C), and the resulting features are globally modeled using a Transformer; the representation vector Z is then reconstructed using a matrix. out Perform linear transformations on each, and then multiply by the weight vector W. Q1 W K1 W V1 We obtain the query vector Q1, the key value vector K1, and the representative value vector V1, as shown in equations (15)-(17):

[0052] Q1 = W Q1 ·LN(Z out (15);

[0053] K1 = W K1 ·LN(Z out (16);

[0054] V1 = W V1 ·LN(Z out (17);

[0055] Where LN represents LayerNorm(), W Q1 W K1 W V1 Let Q1 = (q1, q2, ..., q3) represent the weight matrices of the three newly generated distinct vectors. D1 K1 = (k1, k2, ... k) D1 V1 = (v1, v2, ... v) D1 D1 represents the number of heads in the newly configured multi-head self-attention mechanism;

[0056] Multi-head self-attention is performed on the obtained vectors Q1, K1, and V1; first, the attention Atten is calculated as shown in equation (18):

[0057]

[0058] Where b1 is the newly generated model bias, determined by C 2 *V 2 The system consists of a set of random numbers, representing the uncorrelated encoding of C*V key points.

[0059] The numerical computation of attention Atten is embedded into vector V1, and then combined with vector Z. out Add them together to get Z out1 As shown in equation (19):

[0060] Z out1 =Z out +Atten·V1 (19);

[0061] After multi-head self-attention operations, in order to further capture Z out The characteristics of Z, and then Z out1 The vector Z is obtained through a two-layer feedforward neural network structure. out2 As shown in equation (20):

[0062] Z out2 =Mlp(Z out1 (20).

[0063] The beneficial effects of this invention are:

[0064] 1. This invention processes the captured RGB video using the MMpose method to obtain a skeleton sequence with small data volume, sufficient information, strong robustness and not easily affected by interference. The skeleton sequence is preprocessed and then the GCN method is introduced to process the skeleton data, which eliminates the influence of external information such as lighting, noise, background, and occlusion that RGB video is easily affected by. In addition, the MMpose method can detect and track the posture of multiple people at the same time.

[0065] 2. This invention captures long-term temporal dependencies of the skeleton sequence by using the self-attention mechanism of Transformer, enabling the network to effectively process long-distance dependencies in sequence data, thereby better extracting temporal information;

[0066] 3. This invention uses Transformer to perform global modeling of features, making full use of Transformer's global modeling capabilities. It not only extracts features in the spatial and temporal dimensions, but also performs spatiotemporal fusion of the extracted features in the global dimension, enabling the model to better understand feature information, thereby improving the recognition rate.

[0067] 4. This invention abandons the traditional cross-entropy loss function used for classification and instead uses the focal loss function, which solves the problem of large differences in the recognition results of various actions that are prone to occur in human behavior recognition tasks, thereby improving the overall recognition rate. Attached Figure Description

[0068] Figure 1 This is a network structure diagram of the human behavior recognition method based on GCN and Transformer of the present invention;

[0069] Figure 2 This is a schematic diagram of human body structure segmentation for the human behavior recognition method based on GCN and Transformer of this invention;

[0070] Figure 3 This is a schematic diagram illustrating the human behavior recognition method based on GCN and Transformer of the present invention, which converts RGB video containing human behavior into a skeleton sequence.

[0071] Figure 4 This is a detailed structural diagram of the Transformer in the human behavior recognition method based on GCN and Transformer of this invention.

[0072] Figure 5 This is a graph showing the decrease in training loss for the human behavior recognition method based on GCN and Transformer in this invention. Detailed Implementation

[0073] The present invention will now be described in detail with reference to the accompanying drawings and specific embodiments.

[0074] Example 1

[0075] This invention relates to a human behavior recognition method based on GCN and Transformer, specifically as follows:

[0076] Step 1: Acquire multiple RGB videos containing human movement, extract multiple keyframes from each RGB video, preprocess the extracted keyframe images, crop out the areas containing only human activity, and then label the videos so that each behavior video has a corresponding category label.

[0077] Step 2: Use the human pose estimation algorithm MMPose to process each frame of the image, identify the key points of the human body in the image, arrange the skeleton data in chronological order to form a skeleton sequence, and fill in the sequences of different lengths. Finally, divide the training set and the test set.

[0078] Step 3: Input the training set into the GCN and Transformer networks for training;

[0079] Step 4: Input the test set into the GCN and Transformer networks for testing to achieve human behavior recognition.

[0080] Example 2

[0081] This invention relates to a human behavior recognition method based on GCN and Transformer, such as... Figure 1 As shown, please follow these steps:

[0082] Step 1: Obtain multiple RGB videos containing human movement, extract multiple keyframes from each RGB video, and extract them using the formula shown in Equation (1). Preprocess the extracted keyframe images, manually crop out the areas containing only human activity, and then use the labelimg software to label the videos so that each behavior video has a corresponding category label, thereby reducing the amount of computation and ensuring the consistency of the input data.

[0083] N = T / f (1);

[0084] Where N is the total number of video frames; T is the video duration in seconds; and f is the frame rate.

[0085] Step 2: Use the human pose estimation algorithm MMPose to process each frame of the image, identify the key points of the human body in the image, that is, the position of the joints and the corresponding connecting skeleton, and then fill in the sequences of different lengths.

[0086] Specifically, the human pose estimation algorithm MMPose is used to process each frame of the image, identifying key points of the human body in the image, namely the positions of joints and the corresponding connecting skeletons. For example... Figure 2 As shown, the position of each key point is represented by three-dimensional coordinates (x, y, z). i ,y i ,z i The sequence is represented as ), where i = 1, 2, 3…25; i is the number of keypoints. The skeleton data is then arranged in chronological order to form a skeleton sequence. Typically, the output contains 25 keypoints, such as… Figure 3 As shown, to meet the needs of batch processing and simplify the model training process, padding is typically used to standardize the sequence length, ensuring that the length of all sequences reaches the longest possible length, L. max Then, 70% of the skeleton sequences are used as the training set, and 30% of the skeleton sequences are used as the test set;

[0087] Step 3: Input the training set into the GCN and Transformer networks for training; specifically:

[0088] The GCN extractor used here is PB-GCN, which has good stability and accuracy. GCN enlarges the original data from 3 dimensions to a high-dimensional vector of 256 dimensions. The Transformer structure used in the temporal extractor is the classic Transformer framework, such as... Figure 4 As shown, the Transformer structure used for global modeling is Swin-Transformer.

[0089] Step 3.1: Use the GCN method to extract spatial features from the skeleton sequence corresponding to each keyframe, specifically:

[0090] In an RGB video, the relative coordinates D between various key points within the same frame are... R The relative coordinates D between the same key point in different frames of time. T The vector Z is integrated into the corresponding skeleton sequence in step 2 to form a new representation vector Z. The vector Z is then divided into four parts according to the human skeletal structure: upper limbs, lower limbs, upper trunk and lower trunk. Then, the combination of each joint point of each part is convolved, as shown in equation (2).

[0091]

[0092] Among them, X i (i = 1, 2, 3, 4) represents the combination data for each joint. tj A represents each joint in each combination. i Let D represent the adjacency matrix for each combination. iw represents the degree matrix of each combination. ij This represents the weight of each key point in the combination. After extracting information, a vector X representing the features of each combination is obtained. i Then consider the four vectors X i We perform weighted summation to obtain a higher-level feature vector Z, as shown in equation (3);

[0093]

[0094] Step 3.2, use Transformer to extract the time information of the sequence, specifically:

[0095] Step 3.21: After GCN extracts information, it forms a vector Z with a size of (B, T, C, V), where B represents the batch size, T represents the sequence length, C represents the number of channels, and V represents the number of joints in the skeleton diagram. The dimension of vector Z is transformed to (B*V, T, C), and cosine encoding is performed on each joint of each frame in the skeleton sequence to obtain the order vector PE representing each frame, as shown in equations (4) and (5).

[0096]

[0097]

[0098] Where d represents the dimension of the vector, i.e., C, PE (pos,2i) PE represents the even-numbered dimension of a vector. (pos,2i+1) The odd-numbered dimension of the vector is represented, and each pos represents the absolute position of each frame in the sequence.

[0099] Step 3.22: Add the sequence vector PE to the vector Z to obtain a new representation vector E. From this point on, the vector of each frame contains not only feature information but also sequence information, as shown in equation (6):

[0100] E = Z + PE (6);

[0101] Step 3.23: Perform linear transformations on the representation vector E to obtain the query vector Q, the key value vector K, and the representative value vector V, as shown in equations (7)-(9):

[0102] Q = W Q ·LN(Z) (7);

[0103] K = W K ·LN(Z) (8);

[0104] V = W V ·LN(Z) (9);

[0105] Where LN represents LayerNorm, W Q W K W V Let Q = (q1, q2, ..., q3) represent the weight matrices used to generate three different vectors. D ), K = (k1, k2…k D V = (v1, v2, ... v) D ); D represents the number of heads in the multi-head self-attention mechanism.

[0106] Step 3.3 involves performing multi-head self-attention on the obtained vectors Q, K, and V, specifically as follows:

[0107] Step 3.31, calculate the attention Atten, as shown in Equation (10):

[0108]

[0109] Here, Q is a set of vectors used to find relevant information. The query vector can be viewed as a process of asking questions about each element in the input sequence to determine its correlation with other elements. The key vector K contains the input information at each position, representing each element's response to the query vector Q. The similarity Q·K between the query vector Q and the key vector K is obtained by performing a dot product operation on Q and K. T This can determine the correlation between different positions in the input sequence. d represents the dimension of the query vector Q and the key vector K, and for Q·K... T Divide by The reason is that the vectors need to be normalized. SM stands for softmax(), which transforms the score vector Score between each frame skeleton map into a probability distribution between [0,1], while also highlighting the temporal relationship of the sequence. For Q·K T After softmax, Atten is transformed into a vector with values ​​distributed between [0,1].

[0110] Step 3.32: Embed the attention Atten into the value vector V; finally, add it to the vector Z to obtain the vector Z1, as shown in equation (11):

[0111] Z1=Z+Atten·V (11);

[0112] The value vector contains the input information for each position and is the final object of weighted summation. Compared to the key vector K, which represents the input information, the value vector contains more detailed input information. This is because the K vector is used to calculate similarity, while the V vector is used as the final embedding similarity vector (Atten).

[0113] Step 3.33: To further capture the features of vector Z1, vector Z1 is processed through a two-layer feedforward neural network structure. Specifically, in the first layer of the feedforward network structure, Z1 is linearly transformed. Specifically, vector Z1 is first multiplied by the weight matrix W1, and then the bias vector b1 is added to obtain vector Z2, as shown in equation (12):

[0114] Z2 = W1Z1 + b1 (12);

[0115] Then, the vector Z2 is activated by the ReLU function to capture its nonlinear characteristics, resulting in the vector Z3, as shown in equation (13):

[0116] Z3 = max(0, Z2) (13);

[0117] Finally, the vector Z3 is passed through the second layer of the feedforward neural network. The difference here is that only a linear transformation is performed on the resulting vector Z3; that is, the vector Z3 is first multiplied by the weight matrix W2, and then the bias vector b2 is added to obtain the vector Z. out As shown in equation (14):

[0118] Z out =W2Z3+b2 (14);

[0119] Step 3.4: After extracting features using the GCN and Transformer methods, the resulting vector Z is... out The dimensions are transformed to combine the spatiotemporal features, and the resulting features are modeled globally using Transformer;

[0120] Specifically: This involves vector Z... out The dimensions are then reconstructed to (B, V*T, C), and the resulting features are globally modeled using a Transformer. The representation vector Z is then reconstructed using a matrix. out Perform linear transformations on each, and then multiply by the weight vector W. Q1 W K1 W V1 We obtain the query vector Q1, the key value vector K1, and the representative value vector V1, as shown in equations (15)-(17):

[0121] Q1 = W Q1 ·LN(Z out (15);

[0122] K1 = W K1 ·LN(Z out (16);

[0123] V1 = W V1 ·LN(Z out (17);

[0124] Where LN represents LayerNorm(), W Q1 W K1 W V1 Let Q1 = (q1, q2, ..., q3) represent the weight matrices of the three newly generated distinct vectors. D1 K1 = (k1, k2, ... k) D1 V1 = (v1, v2, ... v) D1 D1 represents the number of heads in the newly configured multi-head self-attention mechanism.

[0125] Multi-head self-attention is performed on the obtained vectors Q1, K1, and V1. First, the attention Atten is calculated, as shown in equation (18):

[0126]

[0127] Where b1 is the newly generated model bias, determined by C 2 *V 2 The expression consists of 3 random numbers, representing uncorrelated encoding of C*V keypoints. SM stands for softmax().

[0128] The numerical computation of attention Atten is embedded into vector V1, and then combined with vector Z. out Add them together to get Z oit1 As shown in equation (19):

[0129] Z out1 =Z out +Atten·V1 (19);

[0130] After multi-head self-attention operations, in order to further capture Z out The characteristics of Z, and then Z out1 The vector Z is obtained through a two-layer feedforward neural network structure. out2 As shown in equation (20):

[0131] Z out2 =Mlp(Z out1 (20);

[0132] Step 3.5: Classify each vector into the category label corresponding to the one in Step 1 using the `nn.Conv2d()` method. Using the statement `nn.Conv2d(256, num_class, kernel_size = 1)`, each vector will have `num_class` numbers. The `softmax()` function converts these numbers into probabilities between 0 and 1. Among the probabilities derived by the network, the probability of the vector actually belonging to a category is p. * Let the video sample be x. i(i = 1, 2, 3…n), where n is the number of samples, according to probability p… * Construct x i The focal loss function is shown in equation (21):

[0133] L(x i )=-(1-p * ) γ log(p * ) (twenty one);

[0134] γ > 0 indicates an adjustable factor. Here, we set γ = 3. After constructing the loss function, the neural network will optimize the parameters constituting the network to reduce the loss function. Here, we set the number of training iterations to 80, the base learning rate of the network to 0.1, and iteratively train and optimize the network hyperparameters using the Adam optimizer and backpropagation algorithm.

[0135] For example, if a video contains 60 categories, there will be 60 different numbers. If 500 videos were recorded, the high-level vectors corresponding to these 500 videos need to be classified into 60 categories. Each vector, after using the `nn.Conv2d()` method, will correspond to 60 numbers, representing the probability that the video belongs to one of the 60 categories. In a computer, these 60 categories are represented by 60 numbers. After obtaining the 60 numbers corresponding to each high-level vector, the `softmax()` function converts these 60 numbers to between 0 and 1, forming 60 probabilities `p`. The highest probability `p` is then found. * p * The corresponding category is the category that the computer considers correct.

[0136] Step 4: Input the test set into the GCN and Transformer networks for testing to achieve human behavior recognition.

[0137] Example 3

[0138] The loss curve obtained by training the network of this invention is shown in the figure. Figure 5 As shown in the table, the training loss gradually decreases with the increase of the number of training rounds, indicating that the model is convergent and effective. Table 1 shows the Top-1 recognition rate of the network on the self-made dataset. It can be seen that, in terms of accuracy, the method of this invention has surpassed other methods.

[0139] Table 1 shows the accuracy on the self-built dataset.

[0140] Lie Group 52.8％ H-RNN 64.0％ Deep LSTM 67.3％ PA-LSTM 70.3％ ST-LSTM+TS 77.7％ Temporal Conv 83.1％ C-CNN+MTLN 84.8％ This invention 87.65％

Claims

1. A human behavior recognition method based on GCN and Transformer, characterized in that, The specific steps are as follows: Step 1: Acquire multiple RGB videos containing human movement, extract multiple keyframes from each RGB video, preprocess the extracted keyframe images, crop out the areas containing only human activity, and then label the videos so that each behavior video has a corresponding category label. Step 2: Use the human pose estimation algorithm MMPose to process each frame of the image, identify the key points of the human body in the image, arrange the skeleton data in chronological order to form a skeleton sequence, and fill in the sequences of different lengths. Finally, divide the training set and the test set. Step 3: Input the training set into the GCN and Transformer networks for training; specifically: Step 3.1: Use the GCN method to extract spatial features from the skeleton sequence corresponding to each keyframe, specifically: In an RGB video, the relative coordinates between various key points within the same frame Relative coordinates between the same key point in different frames of time. The vector Z is then incorporated into the corresponding skeleton sequence in step 2 to form a new representation vector Z. The vector Z is then divided into four parts according to the human skeletal structure: upper limbs, lower limbs, upper trunk, and lower trunk. Convolution is then performed on the combination of each joint point in each part. （2）； in, This represents the combined data for each joint point; This represents each joint in each combination. This represents the adjacency matrix for each combination. This represents the degree matrix for each combination. This represents the weight of each key point in the combination; after extracting information, a vector representing the features of each combination is obtained. Then for the four vectors We perform a weighted summation to obtain a higher-level feature vector Z; （3） Step 3.2: Use Transformer to extract the temporal information of the sequence. Step 3.3: Perform multi-head self-attention operation on the obtained vectors Q, K, and V; Step 3.4: After extracting features using the GCN and Transformer methods, the resulting vector... The dimensions are transformed to combine spatiotemporal features, and the resulting features are then globally modeled using a Transformer; specifically: vector The dimensions are reconverted Then, use Transformer to perform global modeling on the obtained features; reuse the matrix to transform the representation vector. Perform linear transformations separately, and then multiply by the weight vectors respectively. , , , obtain the query vector Key value vector , representing a value vector ; （15）； （16）； （17）； in, This refers to LayerNorm(). , , These represent the weight matrices of the three newly generated different vectors. , , ; This indicates the number of heads in the newly configured multi-head self-attention mechanism; For each of the obtained vectors , , Perform multi-head self-attention operations; first, calculate the attention threshold (Atten). （18）； in, It is the bias of the newly generated model, by Composed of 1 random number, representing C V key points are encoded without correlation; Embed the numerical computation of attentionAtten into a vector Then with vector Adding them together gives ; （19）； After multi-head self-attention operation, in order to further capture The characteristics, and then A vector is obtained through a two-layer feedforward neural network structure. ; （20） Step 3.5: Classify each vector into the category label corresponding to the one in Step 1 using the nn.Conv2d() method; each vector will have num_class numbers, which are converted into probabilities between 0 and 1 using the softmax() function; among the probabilities obtained by the network, the probability that the vector actually belongs to the category is... Let the video sample be... n is the number of samples, according to probability structure The focal loss function is constructed, and after constructing the loss function, the network hyperparameters are optimized through iterative training using the Adam optimizer and backpropagation algorithm. Step 4: Input the test set into the GCN and Transformer networks for testing to achieve human behavior recognition.

2. The human behavior recognition method based on GCN and Transformer as described in claim 1, characterized in that, In step 1, the formula for keyframe extraction is: （1）； Where N is the total number of video frames; T is the video duration in seconds; and f is the frame rate.

3. The human behavior recognition method based on GCN and Transformer as described in claim 1, characterized in that, In step 2, specifically: the human pose estimation algorithm MMPose is used to process each frame of the image to identify the key points of the human body in the image, namely the positions of the joints and the corresponding connecting skeletons, and the position of each key point is expressed in three-dimensional coordinates. It means that among them To determine the number of key points, the skeleton data is then arranged chronologically to form a skeleton sequence. Numerical padding is used to standardize the sequence length, ensuring that the length of all sequences reaches the maximum length of the longest sequence. ; Then, 70% of the skeleton sequences were used as the training set and 30% of the skeleton sequences were used as the test set.

4. The human behavior recognition method based on GCN and Transformer as described in claim 1, characterized in that, In step 3.2, specifically: Step 3.21: After GCN extracts information, it forms vector Z. The dimension of vector Z is then transformed to (B... V, T, C), where B represents the batch size, T represents the sequence length, C represents the number of channels, and V represents the number of joints in the skeleton diagram; cosine encoding is performed on each joint of each frame in the skeleton sequence to obtain the order vector representing each frame. ; （4）； （5）； Where d represents the dimension of the vector, i.e., C. This represents the even-numbered dimension of the vector. The odd-numbered dimension of the vector is represented, and each pos represents the absolute position of each frame in the sequence. Step 3.22, change the order vector Adding it to vector Z yields a new representation vector. This means that the vector of each frame contains not only feature information, but also sequence information. （6）； Step 3.23: Perform linear transformations on the representation vector E to obtain the query vector Q, the key value vector K, and the representative value vector V; （7）； （8）； （9）； in, This indicates LayerNorm. , , These represent the weight matrices used to generate three different vectors. D represents the number of heads in the multi-head self-attention mechanism.

5. The human behavior recognition method based on GCN and Transformer as described in claim 4, characterized in that, In step 3.3, specifically: Step 3.31, calculate the attention Atten; （10）； Here, Q is a set of vectors used to find relevant information. The query vector can be viewed as a process of asking questions about each element in the input sequence to determine its correlation with other elements. The key vector K contains the input information at each position, representing each element in the sequence answering the question to the query vector Q. d represents the dimension of the query vector Q and the key vector K, and SM represents softmax(). Step 3.32: Embed the attention vector Atten into the value vector V; finally, add it to the vector Z to obtain the vector. ; V（11）； Step 3.33, convert the vector Through a two-layer feedforward neural network structure, specifically, in the first layer of the feedforward network structure, for To perform a linear transformation, first transform the vector... Multiply by the weight matrix In addition to the bias vector , to obtain vector ; （12）； Then the vector By using the ReLU activation function, its nonlinear characteristics are captured, resulting in a vector. ; （13）； Finally, the vector Through a second layer of feedforward neural network; first, the vector... Multiply by the weight matrix In addition to the bias vector , to obtain vector ; （14）。 6. The human behavior recognition method based on GCN and Transformer as described in claim 1, characterized in that, In step 3.5, based on probability structure focal loss function; （21）； , indicating an adjustable factor.

Images