Multi-granularity human action classification method based on graph convolutional network

Abstract

A multi-granularity human motion classification method based on graph convolutional networks belongs to the field of video motion recognition. In order to solve the problem of generating accurate and highly robust motion classification results, the key point is to divide the skeleton points into three subsets: the skeleton points themselves, the nearest neighbor nodes close to the center of gravity, and the nearest neighbor nodes far from the center of gravity. On the three channels of the skeleton topology, the input data is processed through convolution, mean compression of different dimensions, and dimensionality expansion to extract the spatial feature matrix and the temporal feature matrix. At the same time, the input data is convolved, etc., which has the effect of accurately classifying both coarse-grained and fine-grained motion data.

Description

Technical Field

[0001] The present invention belongs to the field of video action recognition and relates to a multi-granularity action classification method based on a graph convolutional network. Background Art

[0002] Multi-granularity human action classification based on graph convolutional networks is a very challenging research direction in the field of video action recognition. Among them, the research on the classification method of skeleton point data is the most popular and core research content. Graph convolutional networks are mainly built based on the spatial convolution-temporal convolution mode, and the input information is a two-dimensional or three-dimensional feature sequence.

[0003] At present, research on classification problems in the field of action recognition has made a lot of progress, and various methods have emerged. The data modality studied has expanded from the original RGB data to bone point data, the video frame selection strategy has advanced from the entire input to random multi-frame extraction, and the extracted discriminant information has gradually been refined to the temporal and spatial position information of the human body. Human action classification is aimed at video data of human movement, and the quality of classification performance focuses on the extraction of information on spatial scale and temporal scale. Spatial scale information is manifested as the spatial distribution of human structure in a single video frame, and temporal scale information is manifested as the change of human structure on the time axis. Most of the current mainstream methods only focus on the spatial connection information of nodes in the same frame, and the temporal information is only propagated through time domain convolution. However, these methods ignore the differences between each node in different frames and lose the high-level features at the temporal and spatial levels. Most of the mainstream public data sets are also coarse-grained data sets. In comparison, the research on fine-grained classification problems is relatively slow. Fine-grained classification problems have many practical application scenarios, such as the discrimination of different levels of jumps in skating. Each jump in skating has only slight differences and belongs to a large category of jumps. These fine-grained classification problems are more challenging and more valuable for application. Summary of the invention

[0004] The purpose of the present invention is to provide a multi-granularity human motion classification method based on graph convolutional network, which captures the relationship between skeleton points, between frames and skeleton points, and between frames by extracting the features of time dimension and space dimension in each sample, and then generates accurate and highly robust motion classification results.

[0005] To achieve the above object, the present invention provides the following technical solutions:

[0006] A multi-granularity human action classification method based on graph convolutional network, including

[0007] S1. Obtain the skeleton point dataset of the target action;

[0008] S2. Divide the skeleton points into three subsets: the skeleton point itself, a neighboring node close to the centroid, and a neighboring node far from the centroid. Use V to represent the number of skeleton points. Each subset is a V×V matrix.

[0009] Dividing the adjacency matrix of the three subsets into a shape of 3×V×V according to the three subsets, and dividing the three channels of the skeleton topology according to the adjacency matrix;

[0010] The adjacency matrix is ​​normalized to obtain a skeleton topology matrix A, and Ai is used to represent a subset of the skeleton topology matrix A of the i-th channel;

[0011] S3. Extract the skeleton point data set to obtain the input data X of shape C×T×V in , C represents the number of features, T represents the time window size;

[0012] S4. On the three channels of the skeleton topology, the input data X in By C r Dimensional 1×1 convolution, mean compression and dimension expansion processing of different dimensions are performed to extract the spatial feature matrix X1, X2 and the temporal feature matrix X3. The shape of the spatial feature matrix X1 is C r ×T×1, the shape of the feature matrix X2 of the space is C r ×1×T, the time feature matrix X3 has a shape of C r ×T×1, at the same time, for the input data X in Carry out C out Dimensional 1×1 convolution, resulting in a shape of C out ×T×V space-time vector X4;

[0013] S5. The spatial feature matrix X1 and the spatial feature matrix X2 are modeled by subtraction operation, and the activation function φ1 and C out The 1×1 convolution f1 of dimension generates the spatial vector X5:

[0014] X5=f1(φ1(X1-X2)) (1)

[0015] The spatial feature matrix X2 and the temporal feature matrix X3 are fused through multiplication to establish a spatiotemporal relationship. out The 1×1 convolution f2 of dimension generates the spatiotemporal weight vector X6:

[0016] X6=f2(φ2(X2·X3)) (2)

[0017] S6. Space vector X5 and skeleton topology subset A for each channel i By fusion function Construct the refined space vector X7, space vector X5 and skeleton topology subset A for each channel i The numerical relationship between is learnable, and the weight coefficient is α;

[0018]

[0019] The spatiotemporal weight vector X6 is obtained by the fusion function Provide weights for the space-time vector X4 to generate a refined space-time vector X8;

[0020]

[0021] S7. Perform matrix multiplication and channel concatenation on the two space vectors X7 and X8 on each channel to obtain the spatiotemporal topology.

[0022] S8. Output all channels Add and fuse, and through normalization and ReLu function activation, obtain the channel refinement topology X c ;

[0023] S9. Refine the channel topology X c Input to the time domain convolutional network to get the output X out ;

[0024] S10. Output X out As input data X in Execute steps S4 to S9 several times in a loop, and output X obtained by executing steps S4 to S8 in each loop. out Input the fully connected layer and perform maximum pooling to obtain feature f out , feature f out Represents the action features extracted from the input bone point data, and finally the feature f out Input the Softmax layer to obtain multi-granularity human action classification labels.

[0025] In one embodiment, the output X out As input data X in Steps S4 to S9 are executed eight times in a cycle.

[0026] In one embodiment, the maximum pooling obtains a feature f with a dimension of 256 out .

[0027] In one embodiment, each time domain convolutional network is composed of two time convolution modules, one with a convolution kernel of 3 and the other with a convolution kernel of 5.

[0028] In one embodiment, the network constructed by executing steps S4 to S7 is a spatiotemporal thinning graph convolutional network.

[0029] In one embodiment, the network constructed by executing steps S8 to S9 is a multi-dimensional refined graph convolutional network.

[0030] In one embodiment, the multi-dimensional refined graph convolutional network has nine layers, the first three layers have 64 output channels, the middle three layers have 128 output channels, and the last three layers have 256 output channels. The output of the nine-layer multi-dimensional refined graph convolutional network is input into the fully connected layer, and the maximum pooling is performed to obtain the feature f out .

[0031] In one embodiment, the multi-granularity human action classification method based on graph convolutional network also includes

[0032] S11. Through feature f out And the label y of the training set calculates the chord loss function L AL , and optimize according to the loss function result, iterative training until the preset iteration round is reached; L AL By the conventional loss function L s , angle loss function L in , angle compensation loss function L out Chord length loss function L L Composition: L AL The central feature matrix C is obtained through training. The shape of C is the number of categories K × the number of features F. The feature vector of category y on C is represented by C y ;I L and L out Through the weight constant λ and L s and L in The combined effect gives L AL , expressed by the following formula:

[0033]

[0034]

[0035]

[0036] L AL =L s +L in +λ(L L -L out ) (8)

[0037] Lin makes the distribution within the class more concentrated in terms of angle, L L It makes the distribution within the class more concentrated in terms of distance, L out Increase the distance between classes.

[0038] In one embodiment, the activation function described in step S5 can be any one of tanh, sigmoid, and hardswish.

[0039] In one embodiment, the conventional loss function described in step S11 is any one of Softmax loss, sphereFace, Large margin cosine loss, and arcFace.

[0040] The beneficial effect of the present invention is that on each channel of the skeleton topology, the time dimension and the space dimension are refined at the same time, the high-level features of time and space are felt, and the limitation that the traditional method can only use spatial connection information and time connection information is broken. At the same time, the chord loss function improves the distribution of features in the vector space, making the classification boundaries of different categories clearer. Through the multi-dimensional refined graph convolution mechanism and the chord loss function, both coarse-grained action data and fine-grained action data can be accurately classified, meeting the accuracy requirements in the classification task, and this method has great development prospects. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] Figure 1 Structural diagram of the multi-dimensional refined graph convolution module.

[0042] Figure 2 The structural diagram of a multi-dimensional refined graph convolutional network.

[0043] Figure 3 The structure diagram of the complete multi-dimensional refined graph convolution. DETAILED DESCRIPTION

[0044] Next, we will combine the attached Figures 1 to 3 The specific embodiments of the present invention are described in further detail.

[0045] Example 1: Figures 1 to 3 As shown in Figure 1, a multi-granularity human action classification method based on graph convolutional network includes

[0046] S1: Obtain the skeleton point dataset of the target action through a depth sensor or a skeleton point extraction algorithm, and divide it into a training set T1 and a test set T2;

[0047] S2: Divide the skeleton points into three subsets: the skeleton point itself, a neighboring node close to the center of gravity, and a neighboring node far from the center of gravity;

[0048] Use V to represent the number of bone points. Each subset is a V×V matrix. The division results in an adjacency matrix of shape 3×V×V. The adjacency matrix is ​​then standardized to obtain the skeleton topology matrix A. Different data sets extract different bone key points and have different adjacency relationships, so the obtained adjacency matrices are also different, in order to provide a variety of different types of features for subsequent training.

[0049] In the existing classification method, the input data is input data of the shape of C×t×V, where C is the number of features, t is the total number of frames, the size is not fixed, and V is the number of skeleton points. In the present invention, the adjacency matrix is ​​divided into three channels according to three different subsets, and Ai represents the subset of the skeleton topology matrix A of the i-th channel. S3: Extract frames from the training set T1. The frame extraction strategy is related to the set time window size T and the actual number of frames of the video. The actual number of frames of the video is the total number of frames containing valid data after removing the video frames filled with 0; if the length of the time window is greater than or equal to the actual number of frames, the starting point is randomly selected, the actual video frame is placed as a whole, and the remaining positions are filled with 0; if the length of the time window is smaller than the actual number of frames, the video is divided into time window size intervals, and each interval is randomly extracted, and finally the input data X of the shape of C×T×V is obtained. in The purpose of this step is to fix the time window size of the input data.

[0050] S4: On each channel of the skeleton topology, the three channels corresponding to the three subsets are X in By C r Dimensional 1×1 convolution, mean compression and dimension expansion of different dimensions, extracting spatial feature matrices X1, X2 and temporal feature matrix X3, whose shape is C r ×T×1,C r ×1×T and C r ×T×1At the same time, for X in Carry out C out Dimensional 1×1 convolution, resulting in a shape of C out ×T×V space-time vector X4;

[0051] S5: X1 and X2 are spatially modeled by subtraction operations, and the activation functions φ1 and C out The 1×1 convolution f1 of dimension φ2 generates the spatial vector X5; X2 and X3 are fused through multiplication to establish the spatiotemporal relationship, and the activation function φ2 and C out The 1×1 convolution f2 of dimension generates the spatiotemporal weight vector X6. X5 and X6 are calculated by the following formula;

[0052] X5=f1(φ1(X1-X2)) (1)

[0053] X6=f2(φ2(X2·X3)) (2)

[0054] S6: X5 and each channel's skeleton topology subset A i By fusion function Construct the refined space vectors X7, X5 and A i The numerical relationship between is learnable, and the weight coefficient is α; X6 is Provide weights for the sample's own spatiotemporal vector X4 to generate a refined spatiotemporal vector X8. and It is expressed by the following formula;

[0055]

[0056]

[0057] Each joint point in each frame should have different attention, and different weights should be extracted for different joint points in each frame to generate refined spatiotemporal vectors.

[0058] For multi-granularity series of actions with only one or a few frames of difference, the refined spatiotemporal vector generated by weight allocation can accurately capture subtle movements.

[0059] S7: Perform matrix multiplication and channel concatenation on the two vectors X7 and X8 on each channel to obtain the spatiotemporal topology.

[0060] S8. Output all channels Add and fuse, and through normalization and ReLu function activation, obtain the channel refinement topology X c ;

[0061] S9: X c Input to the time domain convolutional network. Each time domain convolutional network consists of two time convolutional modules with convolution kernels of 3 and 5 to sense the correlation between adjacent frames. The time domain convolutional network can help capture the relationship between adjacent frames and capture the coherent process of the action. Thus, the output X of a layer of multi-dimensional refined graph convolutional network is obtained. out ,From the above, steps S4-S8 are the steps of constructing a multi-dimensional ,refined graph convolutional network.

[0062] S10: will output X out As the input of the next layer of multi-dimensional refined graph convolutional network (X in ), loop through the nine-layer multi-dimensional refined graph convolutional network, that is, loop through steps S4-S9 for a total of 9 rounds, with the input X in is the output X of the previous round out .

[0063] The first three layers have 64 output channels; the middle three layers have 128 output channels; the last three layers have 256 output channels; finally, the output of the nine-layer multi-dimensional refined graph convolutional network is input into the fully connected layer, and the maximum pooling is performed to reduce the feature dimension, and a feature f with a dimension of 256 is obtained. out , represents the action features extracted from the input skeleton point data, and finally the feature f out Input the Softmax layer to obtain multi-granular human action classification labels. The number of layers is selected to achieve a relatively high accuracy, and the efficiency and accuracy are more balanced at 9 layers.

[0064] S10: By f out And the label y of the training set calculates the angular chord loss function L AL , and optimize according to the loss function result, iterative training until the preset iteration round is reached; L AL By the conventional loss function L s , angle loss function L in , angle compensation loss function L out Chord length loss function L L Composition: L AL The central feature matrix C is obtained through training. The shape of C is the number of categories K × the number of features F. The feature vector of category y on C is represented by C y ; L L and L out Through the weight constant λ and L s and L in The combined effect gives L AL , expressed by the following formula.

[0065]

[0066]

[0067]

[0068] L AL =L s +L in +λ(L L -L out ) (8)

[0069] The present invention uses the loss function, Lin, to make the distribution within the class more concentrated in terms of angle, L L It makes the distribution within the class more concentrated in terms of distance, L out Increase the distance between classes.

[0070] Furthermore, the activation function described in the above step S5 can be one of tanh, sigmoid, and hardswish.

[0071] Furthermore, the conventional loss function described in the above step S10 may be one of Softmax loss, sphereFace, Large margin cosine loss, and arcFace.

[0072] Example 2: This example uses a specific figure skating action data set FSD-10 to explain the specific implementation of the method of the present invention in detail. Figures 1 to 3 As shown, the multi-granularity human action classification method based on graph convolutional network of the present invention is specifically implemented as follows:

[0073] S1: Use the openpose25 skeleton point extraction algorithm to extract the skeleton point dataset from the figure skating action dataset FSD-10 and divide it into training set T1 and test set T2.

[0074] S2: Divide the skeleton points into three subsets: the skeleton points themselves, a neighboring node close to the centroid, and a neighboring node far from the centroid, and obtain an adjacency matrix of shape 3×25×25. Then standardize the adjacency matrix to obtain the skeleton topology matrix A.

[0075] S3: Extract frames from T1. The frame extraction strategy and the set time window size T are set to 256. If the length of the time window is greater than or equal to the actual number of frames, the starting point is randomly selected, the actual video frame is put in as a whole, and the remaining positions are filled with 0, and finally the input data X with a shape of 3×256×25 is obtained. in .

[0076] S4: On each channel of the skeleton topology, set X in By C r Dimensional 1×1 convolution, mean compression and dimension expansion of different dimensions, extracting spatial feature matrices X1, X2 and temporal feature matrix X3, whose shape is C r ×25×1, C r ×1×25 and C r ×256×1 at the same time, for X in Carry out C out Dimensional 1×1 convolution, resulting in a shape of C out ×256×25 space-time vector X4;

[0077] S5: X1 and X2 are spatially modeled by subtraction operations, and the activation functions sigmoid and C out The 1×1 convolution f1 of dimension 1 generates the spatial vector X5; X2 and X3 are fused through multiplication to establish the spatiotemporal relationship, and the activation function tanh and C out The 1×1 convolution f2 of dimension 1 generates the spatiotemporal weight vector X6. X5 and X6 are calculated by the following formula.

[0078] X5=f1(sigmoid(X1-X2)) (1)

[0079] X6=f2(tanh(X2·X3)) (2)

[0080] S6: X5 and corresponding channel skeleton topology subset A i By fusion function Construct the refined space vectors X7, X5 and A i The numerical relationship between is learnable, and the weight coefficient is α; X6 is Provide weights for the sample's own spatiotemporal vector X4 to generate a refined spatiotemporal vector X8. and It is expressed by the following formula.

[0081]

[0082]

[0083] S7: Perform matrix multiplication and channel concatenation on the two vectors X7 and X8 on each channel to obtain the spatiotemporal topology. All channels' output Add and fuse, and through normalization and ReLu function activation, obtain the channel refinement topology X c , the complete spatiotemporal refinement process is as follows Figure 1 shown.

[0084] S8: X c Input to the time domain convolutional network, each time domain convolutional network consists of two time convolution modules with convolution kernels of 3 and 5 to sense the correlation between adjacent frames, and obtain the output X of a layer of multi-dimensional refined graph convolutional network out , the process of refining a layer of multi-dimensional graph convolutional network is as follows Figure 2 shown.

[0085] S9: The output is used as the input of the next layer, and a nine-layer multi-dimensional refined graph convolutional network is looped, where the first three layers have 64 output channels; the middle three layers have 128 output channels; the last three layers have 256 output channels; finally, the features are input into the fully connected layer, and the maximum pooling is performed to reduce the feature dimension, and a feature f with a dimension of 256 is obtained. out The complete model structure is as follows Figure 3 shown.

[0086] S10: By f out And the label y of the training set calculates the angular chord loss function L AL , and optimize according to the loss function result, iterative training until the preset iteration round is reached; L ALBy the conventional loss function L s , angle loss function L in , angle compensation loss function L out Chord length loss function L L Composition: L AL The central feature matrix C is obtained through training. The shape of C is 10×256. The feature vector of category y on C is represented by C y ; L L and L out Through the weight constant λ and L s and L in The combined effect gives L AL , λ is set to 0.1, L AL It is expressed by the following formula.

[0087]

[0088]

[0089]

[0090] L AL =L s +L in +λ(L L -L out ) (8)

[0091] S11: Save the training parameters, input the test set T2 into the model, obtain the predicted classification results, use the classification accuracy as the evaluation standard, compare with the label, and compare with other mainstream methods. After repeated tests, the specific results are shown in Table 1.

[0092] Table 1 Test results of the present invention and other comparative algorithms

[0093] algorithm Accuracy Algorithm 1 84.24％ Algorithm 2 88.72％ Algorithm 3 90.58％ Example 1 92.94％ Example 2 93.17％

[0094] Appendix: Algorithm 1: ST-GCN; Algorithm 2: MS-G3D; Algorithm 3: CTR-GCN; Example 1: sigmoid+tanh; Example 2: tanh+tanh;

[0095] Compared with the comparison algorithm, the method corresponding to the second embodiment proposed in the present invention has significantly improved classification accuracy and basically meets the classification accuracy requirements of multi-granularity action data.

[0096] Example 3: The difference between this example and example 1 or 2 is that different activation functions are used, that is, tanh is used for both activation functions φ1 and φ2 in step S5. Example S11: The proposed method is still verified using classification accuracy as an indicator, and the test results are shown in Table 1. Compared with the comparison algorithm, the method corresponding to Example 2 proposed by the present invention has higher accuracy and better meets the classification accuracy requirements of multi-granularity action data.

Claims

1. A multi-granularity human action classification method based on graph convolutional network, characterized in that: include S1. Obtain the skeleton point dataset of the target action; S2. Divide the skeleton points into three subsets: the skeleton point itself, a neighboring node close to the centroid, and a neighboring node far from the centroid. Use V to represent the number of skeleton points. Each subset is a V×V matrix. Dividing the adjacency matrix of the three subsets into a shape of 3×V×V according to the three subsets, and dividing the three channels of the skeleton topology according to the adjacency matrix; The adjacency matrix is ​​normalized to obtain a skeleton topology matrix A, and Ai is used to represent a subset of the skeleton topology matrix A of the i-th channel; S3. Extract the skeleton point data set to obtain the input data X of shape C×T×V in , C represents the number of features, T represents the time window size; S4. On the three channels of the skeleton topology, the input data X in By C r Dimensional 1×1 convolution, mean compression and dimension expansion processing of different dimensions are performed to extract the spatial feature matrix X1, X2 and the temporal feature matrix X3. The shape of the spatial feature matrix X1 is C r ×V×1, the shape of the feature matrix X2 of the space is C r ×1×V, the time feature matrix X3 has a shape of C r ×T×1, at the same time, for the input data X in Carry out C out Dimensional 1×1 convolution, resulting in a shape of C out ×T×V space-time vector X4; S5. The spatial feature matrix X1 and the spatial feature matrix X2 are modeled by subtraction operation, and the activation function φ1 and C out The 1×1 convolution f1 of dimension generates the spatial vector X5: X5=f1(φ1(X1-X2)) (1) The spatial feature matrix X2 and the temporal feature matrix X3 are fused through multiplication to establish a spatiotemporal relationship. out The 1×1 convolution f2 of dimension generates the spatiotemporal weight vector X6: X6=f2(φ2(X2·X3)) (2) S6. Space vector X5 and skeleton topology subset A for each channel i By fusion function Construct the refined space vector X7, space vector X5 and skeleton topology subset A for each channel i The numerical relationship between is learnable, and the weight coefficient is α; The spatiotemporal weight vector X6 is obtained by the fusion function Provide weights for the space-time vector X4 to generate a refined space-time vector X8; S7. Perform matrix multiplication and channel concatenation on the two space vectors X7 and X8 on each channel to obtain the spatiotemporal topology. S8. Output all channels Add and fuse, and through normalization and ReLu function activation, obtain the channel refinement topology X c ; S9. Refine the channel topology X c Input to the time domain convolutional network to get the output X out ; S10. Output X out As input data X in Execute steps S4 to S9 several times in a loop, and output X obtained by executing steps S4 to S8 in each loop. out Input the fully connected layer and perform maximum pooling to obtain feature f out , feature f out Represents the action features extracted from the input bone point data, and finally the feature f out Input the Sofimax layer to obtain multi-granularity human action classification labels.

2. The multi-granularity human action classification method based on graph convolutional network according to claim 1 is characterized in that: Output X out As input data X in Steps S4 to S9 are executed eight times in a loop.

3. The multi-granularity human action classification method based on graph convolutional network according to claim 1 is characterized in that: The maximum pooling obtains the feature f with a dimension of 256 out .

4. The multi-granularity human action classification method based on graph convolutional network according to claim 1 is characterized in that: Each temporal convolutional network consists of two temporal convolutional modules, one with a convolution kernel of 3 and the other with a convolution kernel of 5.

5. The multi-granularity human action classification method based on graph convolutional network according to claim 1 is characterized in that: The network constructed by executing steps S4 to S7 is a spatiotemporal refinement graph convolutional network.

6. The multi-granularity human action classification method based on graph convolutional network according to claim 1 or 5, characterized in that: The network constructed by executing steps S8 to S9 is a multi-dimensional refined graph convolutional network.

7. The multi-granularity human action classification method based on graph convolutional network according to claim 6 is characterized in that: The multi-dimensional refined graph convolutional network has nine layers in total. The first three layers have 64 output channels, the middle three layers have 128 output channels, and the last three layers have 256 output channels. The output of the nine-layer multi-dimensional refined graph convolutional network is input into the fully connected layer, and the maximum pooling is performed to obtain the feature f out .

8. The multi-granularity human action classification method based on graph convolutional network according to claim 1, characterized in that: Also includes S11. Through feature f out And the label y of the training set calculates the angular chord loss function L AL , and optimize according to the loss function result, iterative training until the preset iteration round is reached; L AL By the conventional loss function L s , angle loss function L in , angle compensation loss function L out Chord length loss function L L Composition: L AL The central feature matrix C is obtained through training. The shape of C is the number of categories K × the number of features F. The feature vector of category y on C is represented by C y ; L L and L out Through the weight constant λ and L s and L in The combined effect gives L AL , expressed by the following formula: L AL =L s +L in +λ(L L -L out ) (8) Lin makes the distribution within the class more concentrated in terms of angle, L L It makes the distribution within the class more concentrated in terms of distance, L out Increase the distance between classes.

9. The multi-granularity human action classification method based on graph convolutional network according to claim 1, characterized in that: The activation function described in step S5 can be any one of tanh, sigmoid, and hardswish.

10. The multi-granularity human action classification method based on graph convolutional network according to claim 1, characterized in that: The conventional loss function described in the above step S11 is any one of Sofimax loss, sphereFace, Large margin cosineloss, and arcFace.

Images