Attention spatio-temporal graph convolution network behavior recognition method based on human posture correction

Abstract

The application discloses a kind of based on human posture correction attention spatiotemporal diagram convolution network behavior identification method, comprising the following steps: constructing posture correction module based on convolutional neural network, human posture data is corrected, by adding compensation value to original data, reach the effect of correction;Multiple semantic input module is constructed, and the feature of human posture data after correction is extracted, to get joint node, skeleton information and speed information;The multiple channels corresponding to joint node, skeleton information and speed information are integrated into a road data to carry out feature fusion;Using spatiotemporal diagram convolution network combined with attention mechanism, and introducing residual structure, the integrated feature is trained and tested, and finally the recognition result is obtained by softmax classifier.The application has the beneficial effect that it effectively improves the problem of inaccurate behavior recognition results caused by human posture data errors or missing, thereby improving the accuracy of behavior recognition.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of human behavior recognition, and particularly relates to a human posture correction-based attention spatio-temporal graph convolution network behavior recognition method. BACKGROUND

[0002] With the continuous development of intelligent human-computer interaction technology, the recognition of human behavior has gradually attracted widespread attention in the computer field and the artificial intelligence field. Behavior recognition is an important part in video monitoring, human-computer interaction and other scenes. Through the joint information of the human body, the human behavior can be recognized and predicted, which has important significance for realizing harmonious intelligent human-computer interaction.

[0003] At present, great progress has been made in behavior recognition based on joint information. The mainstream method in the field of behavior recognition is deep learning, including convolutional neural networks, recurrent neural networks and graph convolution networks. In recent years, many methods have achieved good recognition results. However, in actual scenes, the occlusion and offset of human posture data will affect the recognition results to some extent, and affect the accuracy of recognition. SUMMARY

[0004] In order to overcome the influence of occlusion and offset of human posture data on the accuracy of behavior recognition results, the present application provides a human posture correction-based graph convolution network recognition method.

[0005] According to a first aspect of the present application, a human posture correction-based attention spatio-temporal graph convolution network behavior recognition method comprises the following steps:

[0006] S1: constructing a posture correction module based on a convolutional neural network, inputting three-dimensional human posture data, i.e. three-dimensional spatial coordinates (x, y, z), and outputting corresponding compensation values (Δx, Δy, Δz), adding the compensation values to the original data to obtain the corrected human posture data;

[0007] S2: constructing a multi-semantic input module, inputting the corrected human posture data, and obtaining joint, skeleton information and speed information after feature extraction;

[0008] S3: integrating the multi-channel corresponding to the joint, skeleton information and speed information into one piece of data for feature fusion, and obtaining the fused features;

[0009] S4: using a spatio-temporal graph convolution network combined with an attention mechanism, introducing a residual network structure to obtain a residual graph convolution module, inputting the fused features into the residual graph convolution module, training and testing, and obtaining the behavior recognition result through a softmax classifier.

[0010] Preferably, the posture correction module comprises a first convolutional layer, a first graph convolutional layer, a time convolutional layer, a second graph convolutional layer and a second convolutional layer connected in sequence.

[0011] The first convolutional layer and the second convolutional layer have a size of 1*1.

[0012] Preferably, step S1 specifically comprises:

[0013] S1.1: constructing a posture correction module based on a convolutional neural network, inputting three-dimensional human posture data, i.e. three-dimensional spatial coordinates (x, y, z);

[0014] S1.2: learning time-domain features of the change of the joints in the human posture data over time through a time convolutional layer, in a human behavior video sequence, by converting the space-time sequence into a traditional convolutional layer similar to a picture, since the channels remain unchanged, the channels, width and height of the image correspond to the feature frame number, feature number and joint number of the joints, and after each convolution, the next frame is pushed back until the convolution of the entire sequence is completed.

[0015] S1.3: introducing a human joint connection graph to aggregate features through the first graph convolutional layer and the second graph convolutional layer, obtaining a compensation value of the three-dimensional spatial coordinates (x, y, z), and adding the compensation value and the original three-dimensional spatial coordinates (x, y, z) to obtain the corrected human posture data.

[0016] Preferably, step S1.3 specifically comprises:

[0017] In each convolution process, the following formula is used to calculate each joint:

[0018]

[0019] wherein f out (v i ) represents the output feature of the joint v i , f in (v j ) represents the input feature of the joint v j , w(l i (v j )) is a weight function, l i is a mapping that maps the nodes in the neighborhood to their subset labels, B(v i ) represents the set of all joints v i with a distance of 1, i.e. adjacent nodes, and Z ij is a normalization term equal to the number of corresponding subsets; for any joint, the adjacent nodes are the nodes connected in the preset human joint connection graph;

[0020] The input and output of one of the graph convolution layers are represented as:

[0021]

[0022] where f out and f in are the output and input features, respectively, M = ∑ j (A + I), A is an adjacency matrix, I represents self-connection, and W is a weight matrix obtained by all weight convolution, which is obtained by weighted average of nodes through edge weights;

[0023] After the above network processing, the compensation values (Δx, Δy, Δz) of the three-dimensional space coordinates (x, y, z) are obtained, and the compensation values and the original three-dimensional space coordinates (x, y, z) are added to obtain the corrected human posture data V:

[0024] V = (x + Δx, y + Δy, z + Δz).

[0025] Preferably, step S2 specifically comprises:

[0026] S2.1: defining the joint points of the human posture data as follows:

[0027] s = {V i,t |i = 1, 2..., N; t = 1, 2..., T}

[0028] where T is the total number of frames in the sequence, N is the total number of joint points, V i,t represents the joint point i at time t, thereby obtaining the coordinate information corresponding to each joint point of the human body;

[0029] S2.2: according to the obtained human joint point coordinate information, the point close to the center of gravity of the human body in the two joint points is defined as the source joint point, and the coordinate is V i,t = (x i,t , y i,t , z i,t ), and the point far away from the center of gravity of the human body is defined as the target joint point, and the coordinate is V j,t = (x j,t , y j,t , z j,t ), and the length and direction information of the bone are obtained by subtracting the target joint point and the source joint point in the same frame:

[0030] P i,j,t = V j,t -V t,t = (x j,t -x i,t , y j,i -y i,t , z j,t -z i,t )

[0031] S2.3: According to the obtained human body joint coordinate information, the coordinates of the same joint between adjacent frames are subtracted to obtain a speed value representing time sequence information, and the joint coordinate at t frame is defined as V i,t i,t i,t i,t , and the joint coordinate at t+1 frame is defined as V i,t+1 i,t+1 i,t+1 i,t+1 , and the speed information between the joint at t frame and the joint at t+1 frame is represented as:

[0032] Q i,t,t+1 i,t i+1,t i,t i,t+1 i,t i,t+1 i,t i,t+1 .

[0033] Preferably, in step S3, the multi-channel corresponding to the joint, bone information and speed information is integrated into one channel of data for feature fusion, and the specific calculation formula of the fused feature X is:

[0034] X = {V i,t + P i,j,t + Q i,t,t+1 |i = 1, 2,..., N; t = 1, 2,..., T}.

[0035] Preferably, in step S4, the residual graph convolution module specifically comprises: a first branch composed of a first graph convolution network and a time convolution network connected in sequence, and a second graph convolution network connected in parallel with the first branch.

[0036] Preferably, step S4 specifically comprises:

[0037] S4.1: using an attention mechanism to strengthen the processing of the fused feature X to obtain processed feature data;

[0038] ​​​​​​​​​​​​​​S4.2: input the processed feature data into the first graph convolutional network and the second graph convolutional network, train the entire neural network, introduce a cross-domain residual structure to stabilize the training and alleviate gradient descent, each ST-GCN module is allocated to the first graph convolutional network and then to the time convolutional network by the attention mechanism, and the second graph convolutional network is connected in parallel, the same input is maintained in each spatio-temporal graph convolutional module, and the output is adopted in a superimposed manner, realizing the cross-domain function, and this residual connection makes the original layer pay more attention to the changing part of the spatio-temporal information;

[0039] S4.3: obtain the behavior recognition result through the softmax classifier.

[0040] Preferably, step S4.1 specifically comprises:

[0041] S4.1.1: without considering the training batch, the dimension of the fused feature X is CxTxN, C, T and N respectively correspond to the number of channels, the number of frames and the number of key nodes;

[0042] S4.1.2: based on the spatial level attention mechanism, the fused feature X is processed according to the following formula to enhance the key node structure feature of the key part:

[0043] M s =σ(g s (MaxPool t (X))

[0044] Wherein, g s represents one-dimensional convolution in the spatial dimension, MaxPoolt represents maximum pooling in the time dimension, and σ represents the sigmiod activation function;

[0045] S4.1.3: based on the time level attention mechanism, the fused feature X is processed according to the following formula to enhance the frame number important for behavior recognition:

[0046] M t =σ(g t (MaxPool s (X))

[0047] Wherein, g t represents one-dimensional convolution in the time dimension, MaxPools represents maximum pooling in the spatial dimension, and σ represents the sigmiod activation function;

[0048] S4.1.4: the results obtained by the above two attention mechanisms are processed in parallel to obtain the processed feature data M:

[0049] M=M s +M t .

[0050] According to another aspect of the present application, a human posture correction-based attention spatio-temporal graph convolution network behavior recognition device comprises the following modules:

[0051] A posture correction module is configured to construct a posture correction module based on a convolutional neural network, input three-dimensional human posture data, i.e., three-dimensional spatial coordinates (x, y, z), and output corresponding compensation values (Δx, Δy, Δz), which are added to the original data to obtain modified human posture data.

[0052] A feature extraction module is configured to construct a multi-semantic input module, input the modified human posture data, and obtain joint node, bone information and speed information after feature extraction.

[0053] A feature fusion module is configured to integrate the multi-channel corresponding to the joint node, bone information and speed information into one channel of data for feature fusion, and obtain fused features.

[0054] A behavior recognition module is configured to use a spatio-temporal graph convolution network combined with an attention mechanism, introduce a residual network structure to obtain a residual graph convolution module, input the fused features into the residual graph convolution module, and perform training and testing to obtain a behavior recognition result through a softmax classifier.

[0055] Compared with the prior art, the present application has the following advantages:

[0056] The human posture correction-based attention spatio-temporal graph convolution network behavior recognition method and device of the present application correct and optimize human posture data using a human posture correction method, which can compensate for data loss caused by occlusion or deviation, extract multiple types of semantic information, more accurately reflect human behavior, integrate multi-channel input features into one channel of information as input for lightweight processing, reduce the network burden, and finally improve the recognition accuracy through residual graph convolution network combined with an attention mechanism. BRIEF DESCRIPTION OF DRAWINGS

[0057] The present application will be further described below in conjunction with the drawings and embodiments, wherein:

[0058] Figure 1 The flowchart of the human posture correction-based attention spatio-temporal graph convolution network behavior recognition method in the embodiment of the present application;

[0059] Figure 2 The structure diagram of the posture correction module in the embodiment of the present application;

[0060] Figure 3 The human joint node connection diagram in the embodiment of the present application;

[0061] Figure 4 is a feature fusion flowchart in an embodiment of the present application;

[0062] Figure 5 is a residual graph convolution module structure diagram in an embodiment of the present application;

[0063] Figure 6 is a comparison diagram of behavior recognition effects before and after correction in an embodiment of the present application;

[0064] Figure 7 is a structure diagram of the behavior recognition device based on the attention spatio-temporal graph convolution network with human posture correction in an embodiment of the present application. DETAILED DESCRIPTION

[0065] In order to have a clearer understanding of the technical features, objectives and effects of the present application, the specific embodiments of the present application will be described in detail with reference to the accompanying drawings.

[0066] The specific embodiments of the present application will be described in detail below with reference to the accompanying drawings. Figures 1 to 7 Some embodiments disclosed by the present application will be described in detail.

[0067] Embodiment one:

[0068] Please refer to Figure 1 The embodiment provides a behavior recognition method based on the attention spatio-temporal graph convolution network with human posture correction, which comprises the following steps:

[0069] S1: constructing a posture correction module based on a convolutional neural network, inputting three-dimensional human posture data, i.e. three-dimensional spatial coordinates (x, y, z), outputting corresponding compensation values (Δx, Δy, Δz), and adding the compensation values to the original data to obtain the corrected human posture data;

[0070] It should be noted that the human posture data described in the embodiment can also be referred to as human skeleton data or skeleton data in other embodiments, which all represent the same concept and have the same meaning.

[0071] Specifically, step S1 specifically comprises:

[0072] S1.1: constructing a posture correction module based on a convolutional neural network, inputting three-dimensional human posture data, i.e. three-dimensional spatial coordinates (x, y, z);

[0073] Please refer to Figure 2 The posture correction module is constructed with a convolutional neural network, and specifically comprises: a first convolutional layer, a first graph convolutional layer, a time convolutional layer, a second graph convolutional layer and a second convolutional layer connected in sequence;

[0074] The first convolutional layer and the second convolutional layer have a size of 1*1;

[0075] It should be noted that the convolution layer size 1*1 is a default value, which can be adjusted according to actual needs.

[0076] S1.2: learning the time domain features of the joint nodes in the human pose data through the time convolution layer, in a human behavior video sequence, by changing the space-time sequence into a traditional convolution layer similar to a picture, since the channels remain unchanged, the channels, width and height of the image correspond to the feature frame number, feature number and joint number of the joint nodes, and after each convolution, the next frame is pushed back until the entire sequence is completed.

[0077] S1.3: introducing the human joint connection graph through the first graph convolution layer and the second graph convolution layer, as shown in Figure 3 , which includes 17 joint nodes corresponding to different coordinate information, and the compensation value of the three-dimensional space coordinates (x, y, z) is obtained by introducing the human joint connection graph for convolution to aggregate the features, and the compensation value and the original three-dimensional space coordinates (x, y, z) are added to obtain the corrected human pose data.

[0078] More specifically, step S1.3 specifically includes:

[0079] In each convolution process, the following formula is used to calculate each joint node:

[0080]

[0081] wherein f out (v i ) represents the output feature of the joint node v i , f in (v j ) represents the input feature of the joint node v j , w(l i (v j )) is a weight function, l i is a mapping that maps the nodes in the neighborhood to their subset labels, B(v i ) represents the set of all nodes adjacent to the joint node v i , and Z ij is a normalization term, which is equal to the number of corresponding subsets; for any joint node, its adjacent point is the point connected in the preset human joint connection graph;

[0082] The input and output of one graph convolution layer are represented as:

[0083]

[0084] wherein f out and f in are output and input features, respectively, and M=∑ j(A + I), A is an adjacency matrix, I represents self-connection, W is a weight matrix obtained by convolution of all weights, and a node is weighted and averaged by the weight of an edge;

[0085] After the above network processing, a compensation value (Δx, Δy, Δz) of the three-dimensional space coordinates (x, y, z) is obtained, and the compensation value and the original three-dimensional space coordinates (x, y, z) are added to obtain the corrected human pose data V:

[0086] V = (x + Δx, y + Δy, z + Δz).

[0087] S2: Construct a multi-semantic input module, input the corrected human pose data, and obtain joint node, bone information and speed information after feature extraction;

[0088] Specifically, step S2 specifically includes:

[0089] S2.1: The joint nodes of the human pose data are defined as follows:

[0090] s = {V i,t |i = 1, 2..., N; t = 1, 2,..., T}

[0091] Wherein, T is the total number of frames in the sequence, N is the total number of joint nodes, V i,t represents the joint node i at time t, and thus the coordinate information corresponding to each joint node of the human body is obtained;

[0092] S2.2: According to the obtained human joint node coordinate information, the point closer to the center of gravity of the human body in the two joint nodes is defined as the source joint node, and the coordinate is V i,t =(x i,t ,y i,t ,z i,t ), and the point far away from the center of gravity of the human body is defined as the target joint node, and the coordinate is V j,t =(x j,t ,y j,t ,z j,t ). As Figure 3 shown in the figure, joint node 1 is the center of gravity point of the figure, and joint node 2 is the source joint node relative to joint node 3, and joint node 3 is the target joint node of joint node 2. The length and direction information of the bone is obtained by subtracting the target joint node and the source joint node in the same frame:

[0093] P i,j,t =V j,t -V t,t =(x j,t -x i,t ,y j,i -y i,t ,z j,t -z i,t ​

[0094] S2.3: According to the obtained human body joint coordinate information, the coordinates of the same joint between adjacent frames are subtracted to obtain a speed value representing time sequence information, and the joint coordinate at t frame is defined as V i,t i,t i,t i,t i,t+1 i,t+1 i,t+1 i,t+1 Therefore, the speed information between the joint at t frame and the joint at t+1 frame is represented as:

[0095] Q i,t,t+1 i,t i+1,t i,t i,t+1 i,t i,t+1 i,t i,t+1

[0096] S3: The multiple channels corresponding to the joint, bone information and speed information are integrated into one data for feature fusion to obtain fused features;

[0097] Specifically, please refer to Figure 4 In step S3, the multiple channels corresponding to the joint, bone information and speed information are integrated into one data for feature fusion, and the specific calculation formula of the fused features X is:

[0098] X = {V i,t + P i,j,t + Q i,t,t+1 | i = 1, 2,..., N; t = 1, 2,..., T}.

[0099] S4: Using a space-time graph convolution network combined with an attention mechanism, a residual network structure is introduced to obtain a residual graph convolution module, the fused features are input into the residual graph convolution module, training and testing are performed, and a behavior recognition result is obtained through a softmax classifier;

[0100] Specifically, please refer to Figure 5 The residual graph convolution module specifically includes: a first branch composed of a first graph convolution network and a time convolution network connected in sequence, and a second graph convolution network connected in parallel with the first branch.

[0101] Based on the above residual graph convolution module, step S4 specifically includes:

[0102] ​​​​​​​​​​​​​​​​S4.1: using attention mechanism to strengthen the processed fused feature X to obtain processed feature data;

[0103] More specifically, step S4.1 specifically comprises:

[0104] S4.1.1: without considering the training batch, the dimension of the fused feature X is CxTxN, C, T and N respectively correspond to the number of channels, the number of frames and the number of joints;

[0105] S4.1.2: based on the spatial level attention mechanism, the fused feature X is processed according to the following formula to strengthen the key part of the joint structure feature:

[0106] M s =σ(g s (MaxPool t (X))

[0107] Where g s represents a one-dimensional convolution in the spatial dimension, MaxPoolt represents maximum pooling in the time dimension, and σ represents a sigmoid activation function;

[0108] S4.1.3: based on the time level attention mechanism, the fused feature X is processed according to the following formula to strengthen the frame number important for behavior recognition:

[0109] M t =σ(g t (MaxPool s (X))

[0110] Where g t represents a one-dimensional convolution in the time dimension, MaxPools represents maximum pooling in the spatial dimension, and σ represents a sigmoid activation function;

[0111] S4.1.4: the results obtained by the above two attention mechanisms are processed in parallel to obtain processed feature data M:

[0112] M=M s +M t .

[0113] S4.2: input the processed feature data into the first graph convolutional network and the second graph convolutional network, train the entire neural network, introduce a cross-domain residual structure to stabilize the training and relieve gradient descent, such as Figure 4As shown, each ST-GCN (graph convolution and time convolution) module is assigned to the first graph convolution network to the time convolution network by the attention mechanism, and the added second graph convolution network is connected in parallel, the same input is maintained in each spatio-temporal graph convolution module, and the output is adopted in a superimposed manner, realizing the cross-domain function. This residual connection makes the original layer pay more attention to the changing part of the spatio-temporal information;

[0114] S4.3: Obtain the behavior recognition result through the softmax classifier.

[0115] In this embodiment, a human posture correction based attention spatio-temporal graph convolution network behavior recognition method is proposed, and a deep neural network is used to perform human behavior recognition experiments on a sentiment robot. The graph convolution network model is improved, and the experimental results are verified and analyzed. The effect before and after correction is compared as Figure 6 As shown, (a) and (d) are original input pictures, (b) and (e) are pictures obtained before posture correction, and (c) and (f) are pictures obtained after posture correction. It can be seen that the human posture data can be effectively corrected after the recognition method is processed. The feature categories of human posture can be continuously extracted, and the recognition accuracy can be further improved, and the method can be applied to a sentiment robot interaction system.

[0116] Embodiment two:

[0117] Please refer to Figure 6 The embodiment provides a human posture correction based attention spatio-temporal graph convolution network behavior recognition device, which comprises the following modules:

[0118] A posture correction module 1 is used to construct a posture correction module based on a convolutional neural network. The input is three-dimensional human posture data, that is, three-dimensional space coordinates (x, y, z). The output is a corresponding compensation value (Δx, Δy, Δz). The compensation value is added to the original data to obtain the corrected human posture data.

[0119] A feature extraction module 2 is used to construct a multi-semantic input module. The input is the corrected human posture data. After feature extraction, the joint node, bone information and speed information are obtained.

[0120] A feature fusion module 3 is used to integrate the multi-channel corresponding to the joint node, bone information and speed information into one piece of data for feature fusion, and obtain the fused features.

[0121] A behavior recognition module 4 is used to use a spatio-temporal graph convolution network combined with an attention mechanism, introduce a residual network structure, obtain a residual graph convolution module, input the fused features into the residual graph convolution module, and perform training and testing. The behavior recognition result is obtained through a softmax classifier.

[0122] The various modules of the device described in the above embodiment are respectively used to implement the steps of the method described in embodiment one, and can achieve the same technical effects.

[0123] The method and device for behavior recognition based on human posture correction of the embodiment can correct and optimize human posture data by using a human posture correction method, can make up for the problem of data loss caused by occlusion or deviation, can extract multiple types of semantic information, and can more accurately reflect human behavior.

[0124] It should be noted that in this paper, the terms "include", "contain" or any other variants thereof are intended to cover non-exclusive inclusion, so that the process, method, article or system including a series of elements not only includes those elements, but also includes other elements not explicitly listed or inherent to such process, method, article or system. Without more limitations, the element defined by the statement "including a" does not exclude the presence of another identical element in the process, method, article or system including the element.

[0125] The above-mentioned serial numbers of the embodiments of the application are only for description, and do not represent the advantages and disadvantages of the embodiments. In the unit claims of several devices, several of these devices can be embodied by the same hardware item. The use of the words first, second, and third does not represent any order, and these words can be interpreted as identifiers.

[0126] The above is only the preferred embodiment of the application, and does not limit the patent scope of the application, and any equivalent structure or equivalent process transformation using the content of the specification and drawings, or direct or indirect application in other related technical fields, are also included in the patent protection scope of the application.

Claims

1. A human posture correction-based attention spatio-temporal graph convolution network behavior recognition method, characterized in that, The method comprises the following steps: S1: constructing a pose correction module based on a convolutional neural network, inputting three-dimensional human pose data, i.e. three-dimensional spatial coordinates (x, y, z), and outputting corresponding compensation values (Δx, Δy, Δz), adding the compensation values to the original data to obtain corrected human pose data; S2: constructing a multi-semantic input module, inputting the corrected human pose data, and obtaining joint node, bone information and speed information after feature extraction; S3: integrating the multi-channel corresponding to the joint node, bone information and speed information into one channel of data for feature fusion to obtain fused features; S4: using a spatio-temporal graph convolution network combined with an attention mechanism, introducing a residual network structure to obtain a residual graph convolution module, inputting the fused features into the residual graph convolution module for training and testing, and obtaining a behavior recognition result through a softmax classifier; The pose correction module comprises a first convolutional layer, a first graph convolutional layer, a temporal convolutional layer, a second graph convolutional layer and a second convolutional layer connected in sequence; The first convolutional layer and the second convolutional layer have a size of 1*1; Step S1 specifically comprises: S1.1: constructing a pose correction module based on a convolutional neural network, inputting three-dimensional human pose data, i.e. three-dimensional spatial coordinates (x, y, z); S1.2: learning the time domain features of the joint nodes in the human pose data through the temporal convolutional layer, in a human behavior video sequence, the spatio-temporal sequence is converted into a traditional convolutional layer similar to a picture, since the channels remain unchanged, the channels, width and height of the image correspond to the feature frame number, feature number and joint number of the joint nodes, and each convolution is completed, and then a frame is pushed back until the convolution of the entire sequence is completed; S1.3: introducing a human joint node connection graph for convolution to aggregate features through the first graph convolutional layer and the second graph convolutional layer, obtaining compensation values of the three-dimensional spatial coordinates (x, y, z), and adding the compensation values and the original three-dimensional spatial coordinates (x, y, z) to obtain corrected human pose data.

2. The spatio-temporal graph convolutional network for attention method of claim 1, wherein, Step S1.3 specifically comprises: In each convolution process, each joint node is calculated according to the following formula: wherein f out (v i ) represents the output feature of the joint node v i , f in (v j ) represents the input feature of the joint node v j , w(l i (v j )) is the weight function, l i is a mapping that maps the nodes in the neighborhood to their subset labels, B(v i ) represents the set of all nodes that have a distance of 1, i.e. the neighboring nodes, of the joint node v i , and Z ij is a normalization term that equals the number of the corresponding subsets; for any joint node, its neighboring nodes are the nodes that are connected in the pre-set human joint connection graph; The input and output of one graph convolution link are represented as: where f out and f in are output and input features respectively, M = ∑ j (A+I), A is an adjacency matrix, I represents self-connection, and W is a weight matrix obtained by all weight values, which is obtained by weighted average of nodes through edge weights; After the above network processing, the compensation values (Δx, Δy, Δz) of the three-dimensional spatial coordinates (x, y, z) are obtained, and the compensation values and the original three-dimensional spatial coordinates (x, y, z) are added to obtain the corrected human pose data V: V=(x+Δx, y+Δy, z+Δz).

3. The spatio-temporal graph convolutional network for attention method of claim 1, wherein, Step S2 specifically comprises: S2.1: defining the joint nodes of the human pose data as follows: s = {V i,t i = 1,2...,N; t = 1,2,...,T} Wherein, T is the total frame number in the sequence, N is the total joint number, V i,t The joint i at the time t is represented, and thus the coordinate information corresponding to each joint of the human body is obtained. S2.2: According to the obtained human body joint coordinate information, the point close to the human body gravity center in the two joints is defined as the source joint, and the coordinate is V i,t =(x i,t ,y i,t ,z i,t ), and the point far from the human body gravity center is defined as the target joint, and the coordinate is V j,t =(x j,t ,y j,t ,z j,t ), and the length and direction information of the skeleton are obtained by subtracting the target joint and the source joint in the same frame. P i,j,t = V j,t - V i,t = (x j,t - x i,t , y j,t - y i,t , z j,t - z i,t ) S2.3: According to the obtained human body joint coordinate information, the coordinates of the same joint between adjacent frames are subtracted to obtain a speed value representing time sequence information, and the joint coordinate at t frame is defined as V i,t =(x i,t ,y i,t ,z i,t ), the joint coordinate at t+1 frame is defined as V i,t+1 =(x i,t+1 ,y i,t+1 ,z i,t+1 ), and therefore the speed information between the joint at t frame and the joint at t+1 frame is represented as: Q i,t,t+1 = V i,t - V i+1,t = (x i,t - x i,t+1 , y i,t - y i,t+1 , z i,t - z i,t+1 ).

4. The spatio-temporal graph convolutional network attention method for behavior recognition of claim 3, wherein, In step S3, the multi-channel corresponding to the joint node, bone information and speed information is integrated into one channel of data for feature fusion, and the fused features X are calculated according to the following formula: X = {V i,t + P i,j,t + Q i,t,t+i i = 1, 2..., N; t = 1, 2...., T}.

5. The spatio-temporal graph convolutional network for attention method of claim 1, wherein, In step S4, the residual graph convolution module specifically comprises a first branch composed of a first graph convolutional network and a temporal convolutional network connected in sequence, and a second graph convolutional network connected in parallel with the first branch.

6. The spatio-temporal graph convolutional network for attention method of claim 5, wherein, Step S4 specifically comprises: S4.1: using an attention mechanism to enhance the processing of the fused features X to obtain processed feature data; S4.2: input the processed feature data into the first graph convolutional network and the second graph convolutional network, train the whole neural network, introduce a cross-domain residual structure to stabilize the training and relieve gradient descent, each ST-GCN module is assigned to the first graph convolutional network and then to the time convolutional network by the attention mechanism, and the second graph convolutional network is connected in parallel, the same input is maintained in each spatio-temporal graph convolutional module, and the output is obtained in a superimposed manner, thereby realizing the cross-domain function; S4.3: obtain the behavior recognition result through a softmax classifier.

7. The spatio-temporal graph convolutional network attention method for behavior recognition of claim 6, wherein, Step S4.1 specifically includes: S4.1.1: without considering the training batch, the dimension of the fused feature X is C*T*N, C, T and N respectively correspond to the channel number, the frame number and the number of joints; S4.1.2: based on the spatial level attention mechanism, the fused feature X is processed according to the following formula to enhance the joint structure features of the key part: M s = σ(g s (MaxPool t (X)) where g s denotes a one-dimensional convolution in the spatial dimension, MaxPoolt denotes a max-pooling in the temporal dimension, and σ denotes a sigmoid activation function. S4.1.3: based on the time level attention mechanism, the fused feature X is processed according to the following formula to enhance the frame number important to behavior recognition: M t = σ(g t (MaxPool s (X)) where g t denotes a one-dimensional convolution over the time dimension, MaxPools denotes taking the maximum pooling over the spatial dimensions, and σ denotes the sigmoid activation function. S4.1.4: the results obtained by the above two attention mechanisms are processed in parallel to obtain the processed feature data M: M = M s + M t .

8. An attention spatio-temporal graph convolution network behavior recognition device based on human posture correction, characterized in that, It includes the following modules: A pose correction module is configured to construct a pose correction module based on a convolutional neural network, and the input is three-dimensional human pose data, i.e., three-dimensional spatial coordinates (x, y, z), and the output is a corresponding compensation value (Δx, Δy, Δz). The compensation value is added to the original data to obtain the corrected human pose data. A feature extraction module is configured to construct a multi-semantic input module, and the input is the corrected human pose data. After feature extraction, joint, skeleton information and speed information are obtained. A feature fusion module is configured to integrate the multi-channel corresponding to the joint, skeleton information and speed information into one channel of data for feature fusion to obtain fused features. A behavior recognition module is configured to use a spatio-temporal graph convolutional network combined with an attention mechanism, introduce a residual network structure to obtain a residual graph convolutional module, input the fused features into the residual graph convolutional module, and perform training and testing. The behavior recognition result is obtained through a softmax classifier. The pose correction module includes a first convolutional layer, a first graph convolutional layer, a time convolutional layer, a second graph convolutional layer and a second convolutional layer connected in sequence. The first convolutional layer and the second convolutional layer have a size of 1*1. Step S1 specifically includes: S1.1: constructing a pose correction module based on a convolutional neural network, and inputting three-dimensional human pose data, i.e., three-dimensional spatial coordinates (x, y, z); S1.2: learning the time domain features of the joints in the human pose data through the time convolutional layer. In a human behavior video sequence, the time sequence is converted into a traditional convolutional layer similar to a picture. Since the channels remain the same, the channels, width and height of the image correspond to the feature frame number, feature number and joint number of the joint, respectively. After completing each convolution, the next frame is pushed back until the whole sequence is completed. S1.3: introducing a human body joint connection graph through the first graph convolution layer and the second graph convolution layer to aggregate features by convolution, obtaining a compensation value of three-dimensional space coordinates (x, y, z), and adding the compensation value and the original three-dimensional space coordinates (x, y, z) to obtain the corrected human body pose data.

Images