Action recognition method based on ensemble learning method and temporal attention graph convolution

Abstract

The present invention relates to the field of computer recognition technology, and discloses a behavior recognition method based on an integrated learning method fused with a temporal attention graph convolution, comprising the following steps: Step 1: splitting the captured human interaction behavior video into picture frames, each video is split into 148 frames; Step 2: using the Openpose algorithm to realize the data extraction of the three-dimensional coordinate points of the human skeleton points; Step 3: using a graph structure and a joint point feature matrix to describe the human skeleton, and for the graph structure, the graph structure of the human skeleton is constructed in a natural physical connection manner. The present invention is a behavior recognition method based on an integrated learning method fused with a temporal attention graph convolution, and uses a spatiotemporal graph convolution network fused with temporal attention for behavior recognition. In terms of data feature description, a static three-dimensional information flow, an interactive information flow, a motion speed flow, and an interactive motion speed flow are used to describe human features, enriching the semantic information of the features.

Description

Technical Field

[0001] The present invention relates to the field of computer recognition technology, and in particular to a behavior recognition method based on an ensemble learning method and fused with temporal attention graph convolution. Background Art

[0002] With the development of camera technology and the increasing attention paid by the state to social security issues, surveillance cameras are deployed in every corner of society to monitor pedestrian behavior. Video surveillance generates data every minute and every second. How to use computer technology to analyze the behavior data in these videos and correctly identify violent interactive behaviors in order to reduce the consumption of human resources and improve the behavior recognition rate?

[0003] In the prior art, the mainstream behavior recognition methods can be divided into two types: one is based on traditional behavior recognition methods, and the other is based on deep learning. Based on traditional methods, behavior recognition extracts behavior features through manual design and rule-based methods. Representative algorithms include the SURF algorithm based on the Hessian matrix detection operator, the PCA-based non-parametric naive Bayes nearest neighbor (NBNN) classification algorithm, etc. The method based on deep learning uses neural networks to extract features. Representative methods include 3D convolutional neural networks (3D CNN) based on CNN, SCNN networks based on convolution operations and loop operations, SGN networks based on graph convolutions, and ST-GCN networks, etc.

[0004] In traditional RGB video-based human behavior recognition tasks, there are often many factors that affect the task, such as lighting, environmental background, changes in human appearance, etc. In contrast, recognition based on human skeleton joints has the advantages of simple and clear features and is not easily affected by appearance. With the development of posture estimation technologies such as high-precision depth cameras, wearable motion capture systems, and human posture estimation algorithms, we can easily obtain the three-dimensional coordinate information of human joints, so human behavior recognition methods based on graph convolution are widely used in the field of behavior recognition.

[0005] In recent years, human behavior recognition has developed rapidly and achieved many results, but the accuracy is still not high enough. The main problems are as follows: First, the video contains a lot of redundant information that is irrelevant to human behavior, such as background noise, which can easily cause unnecessary interference to recognition; secondly, action is a sequence of information, and actions in different time sequences have different importance for behavior recognition, so how to obtain information on behavior changes in the time dimension and mine the time part that has an impact on behavior recognition is also a research focus; finally, the existing models simply consider the three-dimensional coordinates of the key points of the human skeleton. We can describe the human skeleton from the characteristics of different modes such as interactive information flow and motion speed flow. Multimodal data contains richer semantic features, which can improve the accuracy of behavior recognition. Summary of the invention

[0006] The purpose of the present invention is to provide a behavior recognition method based on an ensemble learning method and a fusion of temporal attention graph convolution, thereby solving the problems raised in the above background.

[0007] To achieve the above object, the present invention provides the following technical solution: a behavior recognition method based on an ensemble learning method and a temporal attention graph convolution, comprising the following steps:

[0008] Step 1: Split the captured human interaction behavior video into picture frames, and split each video into 148 frames;

[0009] Step 2: Use Openpose algorithm to extract the data of three-dimensional coordinate points of human skeleton points;

[0010] Step 3: Use the graph structure and joint feature matrix to describe the human skeleton. For the graph structure, the graph structure of the human skeleton is constructed in a natural physical connection method, and the joint feature matrix is ​​the three-dimensional coordinate tensor of the bone point described in step 2;

[0011] Step 4: For the extraction process of multimodal interaction features, first use the three-dimensional coordinate point feature information of the joint point (stationary three-dimensional information flow) to extract the motion speed flow and the interaction information flow, then obtain the interactive motion speed flow through the motion speed flow, and finally combine the interactive information flow and the interactive motion speed flow into an interactive feature matrix;

[0012] Step 5: Take the graph structure and interaction feature matrix of the human body as the input of the spatiotemporal graph convolution, use the spatial domain graph convolution to extract features in the spatial dimension, and use the temporal graph convolution fused with temporal attention to extract features and mine the image frames that are influential for behavior recognition in the temporal dimension. The spatial domain convolution module and the temporal graph convolution module fused with temporal attention together constitute the spatiotemporal graph convolution module fused with temporal attention;

[0013] Step 6: Repeat step 5 to re-establish a spatiotemporal graph convolution module (in order to create differences in base classifiers, the network structures of the two spatiotemporal graph convolution modules we designed are different) and train it;

[0014] Step 7: Use the two spatiotemporal graph convolution modules established in the previous steps as base classifiers. To avoid differences in training results caused by random division of training data, the training data of each base classifier must be trained and learned through five-fold cross-validation. The training output and test output of each base classifier are integrated separately as the training data input and test input of the meta-classifier.

[0015] Step 8: Use KNN (nearest neighbor algorithm) to perform decision-level data fusion on the prediction result data of the base classifier, and finally realize behavior recognition classification.

[0016] Preferably, in step 2, the Openpose tool is used to process a single frame of the image to extract the three-dimensional coordinate information of the human skeleton joint points in the image. The video contains two human bodies, each human body includes 25 joint points, each joint point includes a 3D coordinate point, and each action contains 148 frames of images. The skeletons of the corresponding picture frames of each video are stacked together in chronological order to form a tensor of size (148, 3, 25, 2), which is the three-dimensional coordinate feature sequence data of the skeleton points.

[0017] Preferably, the graph structure of the human body in step 3 is represented as: G = (v, ε), where v represents the human skeleton joint point, in the present invention, v = 25, and ε represents the edge connected before the skeleton joint point, that is, the human skeleton; for the graph structure, the adjacency matrix G∈R is used. N×N Represents the human skeleton, using the feature matrix X∈R T×C×N×M Represents the characteristic information of human joints; adjacency matrix G∈R N×N The construction rule is: if there is a physical natural connection between human joint point i and human joint point j, then G ij =1, otherwise G ij =0; According to step 2, the feature matrix X is a tensor of size (148, 3, 25, 2).

[0018] Preferably, in step 4, the details of multimodal interaction feature extraction are as follows:

[0019] For a given video skeleton sequence, the formula for its joint points is described as follows:

[0020] S={J mti |m=1,...,M,t=1,...,T,i=1,...,I}(1)

[0021] Where M is the total number of human bodies in a frame of video, T is the total length of the video sequence, I is the total number of joints in the human skeleton, and J is the total number of joints in the human skeleton. mti Describes the joint point i of the mth human body in the video at time t. Before officially starting training, it is necessary to perform diversified preprocessing on the video skeleton sequence S to obtain unimodal interaction data, and fuse different unimodal interaction data to obtain multimodal interaction data. Because the three-dimensional coordinate description of the skeleton joint point can be obtained through a depth camera or a human posture detection algorithm, we give the definition formulas of motion speed flow, interaction information flow, and interactive motion speed flow below;

[0022] Motion speed flow: Motion speed flow is defined as the difference between the same joints of the same human body in two adjacent frames. Since the motion speed difference is the difference between the t-th frame and the t-1-th frame, the calculated motion speed flow will lack the motion speed information of the 0th frame, so the motion speed information of the 0th frame is expanded by filling the matrix with 0 elements. The coordinates of the joint point i in the t-th frame of the m-th human body are defined as:

[0023] J m，t，i =(x m，t，i ,y m，t，i , z m，t，i )

[0024] Then the coordinates of joint point i in the t-1th frame are:

[0025] J m，t-1，i =(x m，t-1，i ,y m，t-1，i , z m，t-1，i )

[0026] Therefore, the joint motion velocity flow of two adjacent frames is expressed as:

[0027] V m，t，i =J m，t，i -J m，t-1，i =

[0028] (x m，t，i -x m，t-1，i ,y m，t，i -y m，t-1，i , z m，t，i -z m，t-1，i ) (2)

[0029] Interactive information flow: Interactive information flow is defined as the difference between the same joints of different human bodies in the same video frame. The coordinates of joint point i in the tth frame of the mth human body are defined as:

[0030] J m，t，i =(x m，t，i ,y m，t，i , z m，t，i )

[0031] Then the coordinates of the joint point i in the tth frame of the nth human body are:

[0032] J n，t，i =(x n，t，i ,y n，t，i , z n，t，i )

[0033] Therefore, the interactive information flow of different human bodies is expressed as:

[0034] M k，t，i =J m，t，i -J n，t，i =

[0035] (x m，t，i -x n，t，i ,y m，t，i -y n，t，i , z m，t，i -z n，t，i ) (3)

[0036] Interactive motion speed flow: Interactive motion speed flow is defined as the difference between the motion speed flows of the same joint points of different human bodies in the same video frame. The motion speed flow of the tth frame of the mth human body is defined as:

[0037] V m，t，i =(α m，t，i , β m，t，i , γ m，t，i )

[0038] Then the motion velocity flow of the nth human body in the tth frame is

[0039] V n，t，i =(α n，t，i , β n，t，i , γ n，t，i )

[0040] Therefore, the interactive motion velocity flow of different human bodies is expressed as:

[0041] MV k，t，i =M m，t，i -M n，t，i =

[0042] (α m，t，i -a n，t，i , β m，t，i -β n，t，i , γ m，t，i -γ n，t，i ) (4)

[0043] The interactive motion velocity flow is calculated by combining equation (2) and equation (4), and the interactive information flow is obtained by combining equation (3). The interactive motion velocity flow and the interactive information flow are stacked and fused to obtain multimodal interactive data. The multimodal interactive data fusion is defined as follows:

[0044] InterFusion = {MV k，t，i +M k，t，i |k=1,...,2;t=1,...,T;i=1,...,I} (5)

[0045] Preferably, in step 5, the spatiotemporal graph convolution module integrating temporal attention can be divided into spatial convolution and temporal convolution, and the module details are as follows:

[0046] In the airspace, the physical connection relationship of the human body is used to compose the image, and the following formula is used to describe the airspace graph convolution:

[0047]

[0048] The setting of α=0.001 is to avoid the existence of 0 values ​​in the adjacency matrix. f represents the input features of the convolutional neural network, W=∑ j w j It is a trainable weight matrix that describes the connection weights and feature importance of the graph connection edges;

[0049] In the temporal graph convolutional network, the SE network module is embedded and the attention mechanism is introduced to mine frames that have an impact on behavior recognition, assign different weights, and improve recognition accuracy;

[0050] The SE module processes the feature map obtained by spatial domain graph convolution to make the model more recognizable for channel features in a specific dimension. It is an attention mechanism. The calculation formula of the SE module is as follows:

[0051]

[0052] S=F ex (z, W)=θ(g(z, W))=θ(W2Relu(W1z)) (8)

[0053] We use formula (7) to encode the spatial features of the time dimension into a global feature by using global average pooling, and then use formula (8) to obtain the weight of each channel in the time dimension. Finally, multiply it with the original feature matrix to obtain a feature matrix with weighted information as the input of the temporal graph convolution. In the time domain, our composition of the time dimension is connected according to the joints corresponding to each frame.

[0054]

[0055] According to formula (9), we can model the joint feature map as a tensor of size C×N×T.

[0056] This tensor is analogous to the RGB feature map of an image. Specifically, C can be compared to the number of channels in the image data, N can be compared to the length of the image data, and T can be compared to the width of the image data. For the above C×N×T feature map, we use K τ ×1 size convolution kernel performs convolution operation on the feature map in the time dimension to extract features, where K τ refers to the convolution kernel size in the time dimension, and the K we use τ is 3.

[0057] Preferably, in step 7 and step 8, the KNN algorithm is used to perform decision-level data fusion on the classification results of the two spatiotemporal graph convolution networks fused with temporal attention. The details are as follows: the training data is cross-divided into five parts, and four parts of the data are used cyclically to train each base classifier model to obtain the classifier model Model ja (j=1,...,2; a=1,...,5). Model ja The model is used to predict the remaining training data and obtain the prediction results. According to formula (10), the five predicted training data are merged to obtain the training data TrainData of the meta-classifier. ja Predict the test data According to formula (11), the test data Perform mean merging to obtain the test data TestData of the meta-classifier.

[0058] Use TrainData as the training data of the KNN algorithm and TestData as the test data to retrain a KNN algorithm model, learn the feature combination obtained by different classifiers, and get the final classification result. (Π in the formula below represents matrix stacking, ∑ in the formula below represents matrix addition)

[0059]

[0060]

[0061] The present invention provides a behavior recognition method based on an ensemble learning method and a time-attention graph convolution. The behavior recognition method based on an ensemble learning method and a time-attention graph convolution has the following beneficial effects:

[0062] (1) The behavior recognition method based on the ensemble learning method and the time-attention graph convolution uses the spatiotemporal graph convolution network fused with time-attention for behavior recognition. In the description of data features, the static three-dimensional information flow, the interactive information flow, the motion speed flow and the interactive motion speed flow are used to describe the human body features, thereby enriching the semantic information of the features.

[0063] (2) The behavior recognition method based on the ensemble learning method and the time-attention graph convolution, in the graph convolution operation, not only mines the change information of the spatial dimension and the time dimension, but also introduces the attention mechanism, calculates the contribution of actions in different time periods to behavior recognition, and mines the behavior frames that have an impact on behavior recognition;

[0064] (3) The behavior recognition method based on the ensemble learning method integrating the temporal attention graph convolution, the present invention uses the KNN algorithm to perform decision-level data fusion on the classification results of two spatiotemporal graph convolutions integrating the temporal attention, so as to achieve the purpose of improving the recognition rate. BRIEF DESCRIPTION OF THE DRAWINGS

[0065] Figure 1 It is a model structure diagram of the present invention;

[0066] Figure 2 The process of obtaining KNN classifier training data from the base classifier;

[0067] Figure 3 The process of obtaining KNN classifier test data from the base classifier;

[0068] Figure 4 The process of using KNN classifier for decision-level data fusion;

[0069] Figure 5 An example of a human skeleton diagram for interactive behavior. DETAILED DESCRIPTION

[0070] like Figure 1-5 As shown, the present invention provides a technical solution: a behavior recognition method based on an integrated learning method and a time attention graph convolution, comprising the following steps:

[0071] Step 1: Split the captured human interaction behavior video into picture frames, and each video is split into 148 frames;

[0072] Step 2: Use the Openpose algorithm to extract the data of the three-dimensional coordinate points of the human skeleton points. Use the Openpose tool to process a single-frame image and extract the three-dimensional coordinate information of the human skeleton joint points in the image. The video contains two human bodies, each of which includes 25 joint points, each of which includes a three-dimensional coordinate point. Each action contains 148 frames of images. The skeletons of the corresponding image frames of each video are stacked together in chronological order to form a tensor of size (148, 3, 25, 2), which is the three-dimensional coordinate feature sequence data of the skeleton points.

[0073] Step 3: Use the graph structure and joint feature matrix to describe the human skeleton. For the graph structure, the graph structure of the human skeleton is constructed in a natural physical connection method, and the joint feature matrix is ​​the three-dimensional coordinate tensor of the bone point described in step 2. The representation of the human graph structure in step 3 is:

[0074] G = (v, ε), where v represents the human skeleton joint point, in the present invention, v = 25, ε represents the edge connected between the skeleton joint points, that is, the human skeleton; for the graph structure, the adjacency matrix G∈R is usedN×N Represents the human skeleton, using the feature matrix X∈R T×N×C×M Represents the characteristic information of human joints; adjacency matrix G∈R N×N The construction rule is: if there is a physical natural connection between human joint point i and human joint point j, then G ij =1, otherwise G ij =0; According to step 2, the feature matrix X is a tensor of size (148, 3, 25, 2);

[0075] Step 4: For the extraction process of multimodal interaction features, first use the three-dimensional coordinate point feature information of the joint points (stationary three-dimensional information flow) to extract the motion speed flow and the interaction information flow, then obtain the interactive motion speed flow through the motion speed flow, and finally combine the interactive information flow and the interactive motion speed flow into an interactive feature matrix. In step 4, the details of the multimodal interaction feature extraction are as follows:

[0076] For a given video skeleton sequence, the formula for its joint points is described as follows:

[0077] S={J mti |m=1,...,M,t=1,...,T,i=1,...,I} (1)

[0078] Where M is the total number of human bodies in a frame of video, T is the total length of the video sequence, I is the total number of joints in the human skeleton, and J is the total number of joints in the human skeleton. mti Describes the joint point i of the mth human body in the video at time t. Before officially starting training, it is necessary to perform diversified preprocessing on the video skeleton sequence S to obtain unimodal interaction data, and fuse different unimodal interaction data to obtain multimodal interaction data. Because the three-dimensional coordinate description of the skeleton joint point can be obtained through a depth camera or a human posture detection algorithm, we give the definition formulas of motion speed flow, interaction information flow, and interactive motion speed flow below;

[0079] Motion speed flow: Motion speed flow is defined as the difference between the same joints of the same human body in two adjacent frames. Since the motion speed difference is the difference between the t-th frame and the t-1-th frame, the calculated motion speed flow will lack the motion speed information of the 0th frame, so the motion speed information of the 0th frame is expanded by filling the matrix with 0 elements. The coordinates of the joint point i in the t-th frame of the m-th human body are defined as:

[0080] J m，t，i =(x m，t，i ,y m，t，i , z m，t，i )

[0081] Then the coordinates of joint point i in the t-1th frame are:

[0082] Jm，t-1 , i=(x m，t-1，i ,y m，t-1，i , z m，t-1，i )

[0083] Therefore, the joint motion velocity flow of two adjacent frames is expressed as:

[0084] V m，t，i =J m，t，i -J m，t-1，i =

[0085] (x m，t，i -x m，t-1，i ,y m，t，i -y m，t-1，i , z m，t，i -z m，t-1，i ) (2)

[0086] Interactive information flow: Interactive information flow is defined as the difference between the same joints of different human bodies in the same video frame. The coordinates of joint point i in the tth frame of the mth human body are defined as:

[0087] J m，t，i =(x m，t，i ,y m，t，i , z m，t，i )

[0088] Then the coordinates of the joint point i in the tth frame of the nth human body are:

[0089] J n，t，i =(x n，t，i ,y n，t，i , z n，t，i )

[0090] Therefore, the interactive information flow of different human bodies is expressed as:

[0091] M k，t，i =J m，t，i -J m，t，i =

[0092] (x m，t，i -x n，t，i ,y m，t，i -y n，t，i , z m，t，i -z n，t，i ) (3)

[0093] Interactive motion speed flow: Interactive motion speed flow is defined as the difference between the same motion speed flows of different human bodies in the same video frame. The motion speed flow of the tth frame of the mth human body is defined as:

[0094] V m，t，i =(α m，t，i , β m，t，i , γm，t，i )

[0095] Then the motion velocity flow of the nth human body in the tth frame is

[0096] V n，t，i =(α n，t，i , β n，t，i , γ n，t，i )

[0097] Therefore, the interactive motion velocity flow between different human bodies is expressed as:

[0098] MV k，t，i =M m，t，i -M n，t，i =

[0099] (α m，t，i -α n，t，i , β m，t，i -β n，t，i , γ m，t，i -γ n，t，t ) (4)

[0100] The interactive motion velocity flow is calculated by combining equation (2) and equation (4), and the interactive information flow is obtained by combining equation (3). The interactive motion velocity flow and the interactive information flow are stacked and fused to obtain multimodal interactive data. The multimodal interactive data fusion is defined as follows:

[0101] InterFusion = {MV k，t，i +M k，t，i |k=1,...,2;t=1,...,T;i=1,...,I} (5)

[0102] Step 5: Take the graph structure and interaction feature matrix of the human body as the input of the spatiotemporal graph convolution, use the spatial domain graph convolution to extract features in the spatial dimension, and use the temporal graph convolution fused with temporal attention to extract features and mine the image frames that are influential for behavior recognition in the temporal dimension. The spatial domain convolution module and the temporal graph convolution module fused with temporal attention together constitute the spatiotemporal graph convolution module fused with temporal attention. In step 5, the spatiotemporal graph convolution module fused with temporal attention can be divided into spatial domain convolution and temporal convolution. The module details are as follows:

[0103] In the airspace, the physical connection relationship of the human body is used to compose the image, and the following formula is used to describe the airspace graph convolution:

[0104]

[0105] The setting of α=0.001 is to avoid the existence of 0 values ​​in the adjacency matrix. f represents the input features of the convolutional neural network, W=∑ j wj It is a trainable weight matrix that describes the connection weights and feature importance of the graph connection edges;

[0106] In the temporal graph convolutional network, the SE network module is embedded and the attention mechanism is introduced to mine frames that have an impact on behavior recognition, assign different weights, and improve recognition accuracy;

[0107] The SE module processes the feature map obtained by spatial domain graph convolution to make the model more recognizable for channel features in a specific dimension. It is an attention mechanism. The calculation formula of the SE module is as follows:

[0108]

[0109] S=F ex (z, W) = θ(g(z, W)) = θ(W2Relu(W1z)) (8) We use formula (7) to encode the spatial features of the time dimension into a global feature by using global average pooling, and then use formula (8) to obtain the weight of each channel in the time dimension. Finally, multiply it with the original feature matrix to obtain a feature matrix with weighted information as the input of the temporal graph convolution. In the time domain, our composition of the time dimension is connected according to the joints corresponding to each frame.

[0110]

[0111] According to formula (9), we can model the joint feature map as a tensor of size C×N×T. This tensor is analogous to the RGB feature map of the image. Specifically, C can be compared to the number of channels in the image data, N can be compared to the length of the image data, and T can be compared to the width of the image data. For the above C×N×T feature map, we use K τ ×1 size convolution kernel performs convolution operation on the feature map in the time dimension to extract features, where K τ refers to the convolution kernel size in the time dimension, and the K we use r is 3;

[0112] Step 6: Repeat step 5 to re-establish a spatiotemporal graph convolution module (in order to create differences in base classifiers, the network structures of the two spatiotemporal graph convolution modules we designed are different) and train it;

[0113] Step 7: Use the two spatiotemporal graph convolution modules established in the previous steps as base classifiers. To avoid differences in training results caused by random division of training data, the training data of each base classifier must be trained and learned through five-fold cross-validation. The training output and test output of each base classifier are integrated separately as the training data input and test input of the meta-classifier.

[0114] Step 8: Use KNN (nearest neighbor algorithm) to perform decision-level data fusion on the prediction result data of the base classifier, and finally realize behavior recognition classification. Use KNN algorithm to perform decision-level data fusion on the classification results of two spatiotemporal graph convolution networks with fusion time attention. The details are as follows: Perform five-fold cross division on the training data, evenly divide it into five parts, and use four parts of data to train each base classifier model in a loop to obtain the classifier model Model ja (j=1,...,2; a=1,...,5). Model ja The model is used to predict the remaining training data and obtain the prediction results. According to formula (10), the five predicted training data are merged to obtain the training data TrainData of the meta-classifier. ja Predict the test data According to formula (11), the test data Perform mean merging to obtain the test data TestData of the meta-classifier.

[0115] Use TrainData as the training data of the KNN algorithm and TestData as the test data to retrain a KNN algorithm model, learn the feature combination obtained by different classifiers, and get the final classification result. (П in the formula below represents matrix stacking, and ∑ in the formula below represents matrix addition)

[0116]

[0117]

[0118] By using a spatiotemporal graph convolutional network that integrates temporal attention for behavior recognition, in terms of data feature description, static three-dimensional information flow, interactive information flow, motion speed flow and interactive motion speed flow are used to describe human body features, enriching the semantic information of the features; in the graph convolution operation, not only the change information of the spatial dimension and the temporal dimension is mined, but also the attention mechanism is introduced to calculate the contribution of actions in different time periods to behavior recognition, and the behavioral frames that have an impact on behavior recognition are mined. Finally, the classification results of the two spatiotemporal graph convolutions that integrate temporal attention are fused at the decision level through the KNN algorithm to achieve the purpose of improving the recognition rate.

Claims

1. An action recognition method based on an ensemble learning method and fused with temporal attention graph convolution, characterized by: The following steps are involved: Step 1: Split the captured human interaction behavior video into picture frames, and split each video into 148 frames; Step 2: Use Openpose algorithm to extract the data of three-dimensional coordinate points of human skeleton points; Step 3: Use the graph structure and joint feature matrix to describe the human skeleton. For the graph structure, the graph structure of the human skeleton is constructed in a natural physical connection method, and the joint feature matrix is ​​the three-dimensional coordinate tensor of the bone point described in step 2; Step 4: For the extraction process of multimodal interaction features, first use the three-dimensional coordinate point feature information of the joint point, which is a static three-dimensional information flow, to extract the motion speed flow and the interaction information flow, then obtain the interactive motion speed flow through the motion speed flow, and finally combine the interactive information flow and the interactive motion speed flow into an interactive feature matrix; Step 5: Take the graph structure and interaction feature matrix of the human body as the input of the spatiotemporal graph convolution, use the spatial domain graph convolution to extract features in the spatial dimension, and use the temporal graph convolution fused with temporal attention to extract features and mine the image frames that are influential for behavior recognition in the temporal dimension. The spatial domain convolution module and the temporal graph convolution module fused with temporal attention together constitute the spatiotemporal graph convolution module fused with temporal attention; Step 6: Repeat step 5 to re-establish a spatiotemporal graph convolution module. The spatiotemporal graph convolution module in step 6 is designed to have a different network structure from that in step 5, and is trained and learned. Step 7: Use the two spatiotemporal graph convolution modules established in the previous steps as base classifiers. To avoid the difference in training results caused by random division of training data, the training data of each base classifier must be trained and learned through five-fold cross validation. The training output and test output of each base classifier are integrated separately as the training data input and test input of the meta classifier. Step 8: Use KNN to perform decision-level data fusion on the prediction result data of the base classifier, and finally realize behavior recognition classification; In step 4, the details of multimodal interaction feature extraction are as follows: For a given video skeleton sequence, the formula for its joint points is described as follows: S={J mti |m=1,...,M,t=1,...,T,i=1,...,I} (1) Where M is the total number of human bodies in a frame of video, T is the total length of the video sequence, I is the total number of joints in the human skeleton, and J is the total number of joints in the human skeleton. mti Described as the joint point i of the mth human body in the video at time t. Before the formal training begins, the video skeleton sequence S needs to be preprocessed in a diversified manner to obtain unimodal interaction data, and different unimodal interaction data are fused to obtain multimodal interaction data. The three-dimensional coordinate description of the skeleton joint point can be obtained through a depth camera or a human posture detection algorithm. The definition formulas of the motion speed flow, interaction information flow, and interactive motion speed flow are as follows: Motion speed flow: Motion speed flow is defined as the difference between the coordinates of the same joint point of the same human body in two adjacent frames. Since the motion speed difference is the difference between the t-th frame and the t-1-th frame, the calculated motion speed flow will lack the motion speed information of the 0th frame. Therefore, the motion speed information of the 0th frame is expanded by filling the matrix with 0 elements. The coordinates of the joint point i in the t-th frame of the m-th human body are defined as: J m，t，i (x m，t，i y m，t，i ,z m，t，i ) Then the coordinates of joint point i in the t-1th frame are: J m，t-1，i -(x m，t-1，i y m，t-1，i ,z m，t-1，i ) Therefore, the joint motion velocity flow of two adjacent frames is expressed as: V m，t，i =J m，t，i -J m，t-1，i =(x m，t，i -x m，t-1，i ,y m，t，i -y m，t-1，i ,z m，t，i -z m，t-1，i ) (2) Interactive information flow: The interactive information flow is defined as the difference between the same joints of different human bodies in the same video frame. The coordinates of the joint point i in the t-th frame of the m-th human body are defined as: J m，t，i (x m，t，i y m，t，i ,z m，t，i ) Then the coordinates of the joint point i in the tth frame of the nth human body are: J n，t，i (x n，t，i y n，t，i ,z n，t，i ) Therefore, the interactive information flow between different human bodies is expressed as: M k，t，i =J m，t，i -J n，t，i =(x m，t，i -x n，t，i ,y m，t，i -y n，t，i ,z m，t，i -z n，t，i ) (3) Interactive motion speed flow: Interactive motion speed flow is defined as the difference between the motion speed flows of the same joints of different human bodies in the same video frame. The motion speed flow of the tth frame of the mth human body is defined as: V m，t，i =(a m，t，i ,b m，t，i ,c m，t，i ) Then the motion velocity flow of the nth human body in the tth frame is V n，t，i =(a n，t，i ,b n，t，i ,c n，t，i ) Therefore, the interactive motion velocity flow of different human bodies is expressed as: MV k，t，i =V m，t，i -V n，t，i =(a m，t，i -a n，t，i ,b m，t，i -b n，t，i ,c m，t，i -c n，t，i ) (4) The interactive motion velocity flow is calculated by combining equation (2) and equation (4), and the interactive information flow is obtained by combining equation (3). The interactive motion velocity flow and the interactive information flow are stacked and fused to obtain multimodal interactive data. The multimodal interactive data fusion is defined as follows: InterFusion={MV k，t，i +M k，t，i |k=1,...,2;t=1,...,T;i=1,...,I} (5)。 2. The behavior recognition method based on ensemble learning method and temporal attention graph convolution according to claim 1 is characterized by: In step 2, the Openpose tool is used to process a single frame of the image and extract the three-dimensional coordinate information of the human skeleton joints in the image. The video contains two human bodies, each of which includes 25 joints, each of which includes a three-dimensional coordinate point. Each action contains 148 frames of images. The skeletons of the corresponding image frames of each video are stacked together in chronological order to form a tensor of size (148, 3, 25, 2), which is the three-dimensional coordinate feature sequence data of the skeleton points.

3. The behavior recognition method based on ensemble learning method and temporal attention graph convolution according to claim 1 is characterized by: The graph structure of the human body in step 3 is represented as: G = (v, ε), where v represents the human skeleton joints, v = 25, and ε represents the edges connecting the skeleton joints, that is, the human skeleton; for the graph structure, use the adjacency matrix G∈R N×N Represents the human skeleton, using the feature matrix X∈R T×C×N×M Represents the characteristic information of human joints; Adjacency matrix G∈R N×N The construction rule is: if there is a physical natural connection between human joint point i and human joint point j, then G ij =1, otherwise G ij =0; According to step 2, the feature matrix X is a tensor of size (148, 3, 25, 2).

4. The behavior recognition method based on ensemble learning method and temporal attention graph convolution according to claim 1 is characterized in that: In step 5, the spatiotemporal graph convolution module integrating temporal attention can be divided into spatial convolution and temporal convolution. The module details are as follows: In the airspace, the physical connection relationship of the human body is used to compose the image, and the following formula is used to describe the airspace graph convolution: The setting of α=0.001 is to avoid the existence of 0 values ​​in the adjacency matrix. f represents the input features of the convolutional neural network. W=∑ j w j It is a trainable weight matrix that describes the connection weights and feature importance of the graph connection edges; In the temporal graph convolutional network, the SE network module is embedded and the attention mechanism is introduced to mine frames that have an impact on behavior recognition, assign different weights, and improve recognition accuracy; The SE module processes the feature map obtained by spatial domain graph convolution to make the model more recognizable for channel features in a specific dimension. It is an attention mechanism. The calculation formula of the SE module is as follows: S=F ex (z,W)=θ(g(z,W))=θ(W2Relu(W1z)) (8) Using formula (7), the spatial features of the time dimension are encoded into a global feature by using global average pooling. Then, using formula (8), the weight of each channel in the time dimension is obtained. Finally, it is multiplied with the original feature matrix to obtain a feature matrix with weighted information as the input of the time graph convolution. In the time domain, the composition of the time dimension is connected according to the joints corresponding to each frame. According to formula (9), the joint feature map is modeled as a tensor of size C×N×T. This tensor is analogous to the RGB feature map of the image. Specifically, C can be compared to the number of channels in the image data, N can be compared to the length of the image data, and T can be compared to the width of the image data. The above C×N×T feature map is modeled as a tensor of size C×N×T using K τ ×1 size convolution kernel performs convolution operation on the feature map in the time dimension to extract features, where K τ Refers to the convolution kernel size in the time dimension, and the K used τ is 3.

5. The behavior recognition method based on ensemble learning method and temporal attention graph convolution according to claim 1 is characterized in that: In steps 7 and 8, the KNN algorithm is used to perform decision-level data fusion on the classification results of the two spatiotemporal graph convolution networks fused with temporal attention. The details are as follows: the training data is divided into five parts evenly by five folds, and four parts of the data are used to train each base classifier model in a loop to obtain the classifier model Model ja (j=1,...,2; a=1,...,5), Model ja The model is used to predict the remaining training data and obtain the prediction results. According to formula (10), the five predicted training data are merged to obtain the training data TrainData of the meta-classifier. Similarly, we can use Model ja Predict the test data According to formula (11), the test data Perform the mean method to merge and obtain the test data TestData of the meta-classifier. Use TrainData as the training data of the KNN algorithm and TestData as the test data. Retrain a KNN algorithm model to learn the feature combination obtained by different classifiers and obtain the final classification result. Π in the following formula represents matrix stacking, and ∑ in the following formula represents matrix addition:

Images