Multi-scale image convolutional network adaptive integration system and method for human skeleton action recognition

Abstract

The invention provides a multi-scale image convolutional network adaptive integration system and method for human skeleton action recognition, and the method comprises the steps: S0, obtaining human skeleton data, and constructing a multi-scale hierarchical decomposition image according to the human skeleton data; s1, performing feature extraction and classification on the human body skeleton data based on the multi-scale hierarchical decomposition graph; and S2, performing fusion operation on the multiple groups of preliminary action classification results of the human body skeleton data to obtain a group of action classification results of the human body skeleton data. The system comprises a multi-scale image convolutional network and a self-adaptive integrated network which are connected in sequence, wherein the multi-scale image convolutional network comprises a human body overall feature map convolutional network, a human body local feature map convolutional network and a human body detail feature map convolutional network which are parallel. According to the method, the classification precision is improved, manual intervention is reduced, and the influence of unreasonable weight parameter setting on the model precision is weakened.

Description

Technical Field

[0001] The present invention relates to the technical field of action recognition, and in particular to a system and method for human skeleton action recognition. Background Art

[0002] Human action recognition is an important research direction in the field of computer vision and artificial intelligence. It has wide applications in many fields such as intelligent monitoring, human-computer interaction, virtual reality, motion analysis, and health monitoring. In the study of human action recognition, skeleton-based action recognition is a popular method. Compared with the action recognition method based on RGB images, the skeleton-based method has the advantages of strong anti-interference ability, low computational complexity, and strong adaptability. With the development of deep learning, skeleton action recognition methods based on graph convolutional networks (GCNs) have received widespread attention. In this method, the human skeleton is modeled as a graph structure, where nodes represent the joints of the human body and edges represent the relationship or connection between joints.

[0003] Graph convolutional networks can effectively model and classify human actions using graph-structured data. In early human skeleton action recognition methods based on graph convolutional networks, the human physical connection graph focused on expressing the connection between human joints, and the graph structure remained fixed. Some subsequent methods learned the relationship between joints through attention dynamics, but because they used the human physical connection graph as the learning graph, the human physical connection occupied a dominant position, so they could not learn the relationship between some joints well, especially the relationship between joints that were far away. The recent hierarchical decomposition graph convolution solves this problem well by replacing the physical connection edges with fully connected edges, and obtains hierarchical decomposition graphs with different structures by selecting different human center of gravity points, and then extracts features with complementary relationships. However, the human center of gravity point is a global key point. The graph structure constructed from this point is extended layer by layer to the whole body, capturing the overall movement trend of the human body, and cannot capture the movement of local or detailed parts of the human body. This means that it may not perform well in some scenes that need to capture local or subtle dynamics, thereby affecting the classification accuracy of human skeleton action recognition tasks.

[0004] Multi-stream network integration is an effective method to improve the performance of models in complex tasks by integrating diverse features, enhancing robustness and improving accuracy. It is also very common in human skeleton action recognition tasks based on graph convolution. Some skeleton action recognition methods use multi-stream structures due to different forms of input data. They divide the form of input data into skeleton point form, skeleton form, dynamic skeleton point form, dynamic skeleton form or other forms. Some skeleton action recognition methods use multi-stream structures due to different graph structures, such as constructing different graph structures through different human body center of gravity points in layered decomposition graph convolution to extract different features. However, these multi-stream networks for human skeleton action recognition mostly use simple averaging methods when integrating, and sometimes use weighted averaging methods with artificially given weights. These methods are all linear fusions of multi-stream network features, and the weights of linear fusion are artificially given hyperparameters, that is, both the fusion method and the fusion parameters are subjectively given by the researchers, which may lead to instability and unpredictability of the results. In addition, the weight of each stream is unchanged during the entire fusion process, and cannot be dynamically adjusted for different action types. The way multi-stream network features are fused will have a great impact on the results, so how to effectively fuse multi-stream network features is also a question worth exploring. Summary of the invention

[0005] In view of the technical problems that the prior art performs poorly in certain scenarios that need to capture local or subtle dynamics, resulting in limited recognition accuracy, and cannot fully integrate multi-stream network features, resulting in low classification accuracy, the present invention proposes a multi-scale graph convolutional network adaptive integration system and method for human skeleton action recognition. The multi-scale graph convolutional network is used to extract the overall, local and subtle multi-scale motion features of the human body, and the output results of the multi-scale graph convolutional network are integrated through an adaptive integration network to obtain a more accurate classification result, thereby improving the recognition accuracy of human skeleton action.

[0006] In order to achieve the above object, the technical solution of the present invention is achieved as follows:

[0007] A multi-scale graph convolutional network adaptive integration method for human skeleton action recognition, characterized by comprising the steps of:

[0008] S0: Obtain human skeleton data and construct a multi-scale hierarchical decomposition graph based on the human skeleton data;

[0009] S1: Extract features from human skeleton data and classify them based on a multi-scale hierarchical decomposition graph to obtain multiple groups of preliminary action classification results for human skeleton data;

[0010] S2: Perform a fusion operation on multiple groups of preliminary action classification results of the human skeleton data to obtain a group of action classification results of the human skeleton data.

[0011] The human skeleton data includes a bone point form and a skeleton form;

[0012] The multi-scale hierarchical decomposition diagram includes a human body overall hierarchical decomposition diagram, a human body local hierarchical decomposition diagram and a human body detail hierarchical decomposition diagram;

[0013] The human body overall layered decomposition diagram is a single-point layered decomposition diagram starting from the joint point of the human body's center of gravity;

[0014] The human body local layered decomposition diagram is a double-point layered decomposition diagram starting from the symmetrical points of the limbs;

[0015] The human body detail layered decomposition diagram is a single-point layered decomposition diagram to which the hand and upper limb-head connection diagram structure is added based on the human body overall layered decomposition diagram.

[0016] The method for obtaining the double-point layered decomposition diagram is as follows: select the initial joint point n of the symmetric relationship 1 and the initial joint point n 2 , respectively, from the initial joint point n 1 and the initial joint point n 2 Starting from two points, the human skeleton is decomposed into single points and layers, and two sets of single point decomposition point sets are obtained. and Then merge the two single point decomposition point sets and Get the set of double point decomposition points Single point decomposition point set Single point decomposition point set and the set of double point decomposition points Contains N L point sets; finally, the point set is decomposed according to the double point The adjacency matrix of the double-point hierarchical decomposition graph is constructed using the method of constructing the single-point hierarchical decomposition graph.

[0017] The human body center of gravity joints include hip joints, abdomen joints and chest joints;

[0018] The symmetrical points of the limbs include left and right shoulder joints, left and right elbow joints, left and right hip joints, and left and right knee joints;

[0019] The hand and upper limb-head connection graph structure includes a finger joint point set and an upper limb-head joint point set, wherein the upper limb-head joint point set includes a wrist joint, an elbow joint and a point representing the head.

[0020] The method for extracting features from human skeleton data and classifying them based on a multi-scale hierarchical decomposition graph is as follows: an adjacency matrix of the human body overall hierarchical decomposition graph and an adjacency matrix of the human body detail hierarchical decomposition graph are obtained from the human body overall hierarchical decomposition graph and the human body detail hierarchical decomposition graph respectively, and an adjacency matrix of the human body local hierarchical decomposition graph is obtained from the human body local hierarchical decomposition graph using the method for obtaining the double-point hierarchical decomposition graph; then multi-layer graph convolution operations are performed on the adjacency matrix of the human body overall hierarchical decomposition graph, the adjacency matrix of the human body detail hierarchical decomposition graph, and the adjacency matrix of the human body local hierarchical decomposition graph respectively, and then a plurality of groups of preliminary action classification results corresponding to the human body overall hierarchical decomposition graph, a plurality of groups of preliminary action classification results corresponding to the human body detail hierarchical decomposition graph, and a plurality of groups of preliminary action classification results corresponding to the human body local hierarchical decomposition graph are obtained by a fully connected layer.

[0021] The fusion operation in step S2 is specifically as follows:

[0022] For each type of action, define a regression model:

[0023] P i =f(r 1 , r 2 , ..., r i ,...,r S )

[0024] Among them, r i is a set of preliminary action classification results, P i It indicates the score of the action sample belonging to the i-th action, and the predicted score of the j-th action sample on the i-th action is P ij ∈P i ;

[0025] The regression model is used to perform multiple regression calculations on multiple groups of preliminary action classification results of human skeleton data to obtain the prediction score of each action sample; the maximum value is judged based on the prediction score of each action sample: if it satisfies Then the determined action type of the jth action sample is the I′th action.

[0026] A multi-scale graph convolutional network adaptive integration system for human skeleton action recognition, comprising an input network, a multi-scale graph convolutional network and an adaptive integration network connected in sequence;

[0027] The input network is used to obtain human skeleton data and construct a multi-scale hierarchical decomposition graph based on the human skeleton data;

[0028] The multi-scale graph convolutional network extracts features from the human skeleton data and classifies them based on the multi-scale hierarchical decomposition graph to obtain multiple groups of preliminary action classification results for the human skeleton data;

[0029] The adaptive integrated network is used to perform a fusion operation on preliminary action classification results of multiple groups of human skeleton data to obtain an action classification result of a group of human skeleton data.

[0030] The multi-scale graph convolution network includes a parallel human body overall feature graph convolution network, a human body local feature graph convolution network and a human body detail feature graph convolution network;

[0031] The human body overall feature map convolutional network is used to obtain a human body overall hierarchical decomposition map, and multi-level feature extraction and classification are performed according to the human body overall hierarchical decomposition map;

[0032] The human body local feature map convolution network is used to obtain a human body local hierarchical decomposition map, and perform multi-level feature extraction and classification according to the human body local hierarchical decomposition map;

[0033] The human body detail feature map convolutional network is used to obtain a human body detail hierarchical decomposition map, and perform multi-level feature extraction and classification based on the human body detail hierarchical decomposition map.

[0034] The human body overall feature graph convolution network includes 3 parallel graph convolution networks; the human body local feature graph convolution network includes 4 parallel graph convolution networks; the human body detail feature graph convolution network includes 3 parallel graph convolution networks;

[0035] The graph convolutional network uses different hierarchical decomposition graph structures for feature extraction.

[0036] The adaptive integrated network includes an input layer, a hidden layer and an output layer connected in sequence;

[0037] The input layer is used to receive multiple groups of preliminary action classification results of human skeleton data output by the multi-scale graph convolutional network;

[0038] The hidden layer is used to perform multivariate regression calculation on multiple groups of preliminary action classification results of human skeleton data to obtain the predicted probability of each action sample; the hidden layer includes a fully connected layer I, a ReLU activation function layer and a fully connected layer II connected in sequence;

[0039] The output layer is used to determine the maximum value based on the predicted probability of each action sample, and output the action sample and its corresponding determined action type.

[0040] The beneficial effects of the present invention are as follows: the present invention extracts the overall, local and fine multi-scale motion features of the human body through a multi-scale graph convolutional network, enhances the performance of the model in certain scenarios that need to capture local or subtle dynamics, and the rich and diverse features will also improve the overall performance of the model; through an adaptive integrated network, the features obtained by the multi-scale graph convolutional network are fully utilized and fully integrated, so that the most effective feature combination can be selected for different types of actions, which not only improves the classification accuracy of the model, but also reduces manual intervention and weakens the impact of unreasonable weight parameter settings on the model accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0042] Figure 1 Flow chart of the method of the present invention.

[0043] Figure 2 It is a schematic diagram of the overall structure of the present invention.

[0044] Figure 3 This is the human skeleton joint point map of the NTU-RGB+D 60 dataset.

[0045] Figure 4 This is an example of the hierarchical decomposition graph structure of the convolutional network of the human body overall, local and detail feature graphs of the present invention.

[0046] Figure 5 This is a graph convolutional network structure diagram for each stream of the present invention.

[0047] Figure 6 This is the structural diagram of the adaptive integration algorithm of the present invention. DETAILED DESCRIPTION

[0048] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0049] Embodiment 1:

[0050] The human skeleton data selected in Example 1 comes from the NTU-RGB+D 60 dataset, which contains 60 types of actions. The human skeleton in the dataset consists of 25 joint points, such as Figure 3 As shown, the following are the specific steps.

[0051] A multi-scale graph convolutional network adaptive integration method for human skeleton action recognition, such as Figure 1 As shown, the steps are:

[0052] S0: Obtain human skeleton data and construct a multi-scale hierarchical decomposition graph based on the human skeleton data;

[0053] The human skeleton data is represented as a graph in is the node set, ε represents the edge set, represents the joints of the human body (such as wrists, elbows, knees, etc.), and ε represents the connection between these joints. For example, there is an edge between the wrist and elbow joints on the arm. The corresponding 3D skeleton sequence data is represented as T and V represent the number of frames and joint points respectively, X[:,t i ,v i ]∈{x,y,z} represents the vth i The joint point is at the tth i 3D coordinates at frame time.

[0054] There are two forms of human skeleton data, namely bone point form and skeleton form. The bone point form is the coordinates of the joint points in the original skeleton, and the skeleton form is the spatial difference between the coordinates of the joint points, that is, the coordinate distance.

[0055] The multi-scale hierarchical decomposition diagram includes a human body overall hierarchical decomposition diagram, a human body local hierarchical decomposition diagram and a human body detail hierarchical decomposition diagram, so as to effectively extract different degrees of motion features of the human skeleton.

[0056] The human body overall layered decomposition diagram is a single-point layered decomposition diagram starting from the joint point of the human body's center of gravity, and the classification result obtained from it can reflect the overall movement trend of the human body;

[0057] The method for obtaining the single-point hierarchical decomposition diagram is as follows: first, the human skeleton containing V joints needs to be decomposed layer by layer from the inside to the outside in the order of physical connections of the human body. Specifically, starting from the initial joint point, the initial joint point is the first-layer single-point decomposition point set H 1 , the joint points connected to the center of gravity joint points in the human body physical connection are the second-level single-point decomposition point set H 2 , the physical connection between the human body and the second-layer single-point decomposition point set H 2All the joint points connected to each joint point in the third layer form a single point decomposition point set H 3 , and so on, until all joints are decomposed, we will eventually get N H single point decomposition point set, represented as a single point decomposition point set set Among them, the single-point decomposition point set of the latter layer does not include the joint points that appeared in the single-point decomposition point set of the previous layer.

[0058] Then construct the adjacency matrix of the single-point hierarchical decomposition graph representing The table is:

[0059]

[0060] ε k =ε(H k ∪H k+1 , H k →H k+1 , H k+1 →Hx)

[0061] Where N HL Represents the number of layers in the hierarchical decomposition diagram, N HL =N H -1, H k represents the k-th level single point decomposition point set; ε k It means S = {s id ,s cp ,s cf}, N S =3 represents three edge subsets; s id =ε(H k ∪H k+1 ) represents the single point decomposition point set H k , H k+1 The subset of edges that connect the middle node to itself; s cp =ε(H k →H k+1 ) represents the decomposition of the point set H from a single point k Decomposition of point set H from central node to single point k+1 The subset of centrifugal edges connected to the central node, the two single-point decomposition point sets are connected in the form of full connection, that is, the single-point decomposition point set H k All the related nodes are respectively related to the single point decomposition point set H k+1 All relevant nodes are connected; cf =ε(H k+1 →H k ) represents the decomposition of the point set H from a single point k+1 Decomposition of point set H from central node to single point k The centripetal edge subset connected to the midpoint node, the two single-point decomposition point sets are connected in the form of full connection, that is, the single-point decomposition point set Hk+1 All the related nodes are respectively related to the single point decomposition point set H k All relevant nodes are connected.

[0062] The local layered decomposition diagram of the human body is a double-point layered decomposition diagram starting from the symmetrical points of the limbs, which can reflect the movement of the local structure of the human body, that is, the situation of the limbs of the human body, which are the parts of the human body that move faster;

[0063] The method for obtaining the double-point layered decomposition diagram is as follows: select the initial joint points n in a symmetrical relationship 1 and the initial joint point n 2 According to the joint point decomposition method in the single-point layered decomposition graph acquisition method, the initial joint point n 1 and the initial joint point n 2 Starting from two points, the human skeleton is decomposed to obtain two single-point decomposition point sets and Single point decomposition point set and Contains N L single point decomposition point set, single point decomposition point set The j-th single-point decomposition point set is Single point decomposition point set The j-th single-point decomposition point set is n is the number of joint points in the point set, and then merge the two single-point decomposition point sets and Get the set of double point decomposition points Double point decomposition point set The j-th double-point decomposition point set in is Double point decomposition point set Contains N L The final set of double-point decomposition points is Finally, the adjacency matrix representing the two-point hierarchical decomposition graph is constructed The method is the same as the construction method in the above single-point layered decomposition diagram.

[0064] The human body detail layered decomposition diagram is a single-point layered decomposition diagram that adds a hand and upper limb-head connection hierarchical diagram structure to the human body overall layered decomposition diagram. The hand is the most flexible part of the human body and can reflect more detailed movement conditions.

[0065] The acquisition method is:

[0066] First, construct the adjacency matrix of the single-point hierarchical decomposition graph starting from the center of gravity joint point It is expressed as:

[0067]

[0068] Then construct the hand and upper limb-head connection hierarchical graph structure;

[0069] The hierarchical graph structure of the hand-upper limb-head connection includes: a finger joint point set F and an upper limb-head joint point set L, wherein the upper limb-head joint point set includes a wrist joint, an elbow joint and a point representing the head;

[0070] The hierarchical graph structure of the hand and upper limb-head connection is represented as:

[0071]

[0072] Finally, the adjacency matrix representing the hierarchical decomposition graph of the human body is constructed It is expressed as:

[0073]

[0074] S1: Feature extraction and classification of human skeleton data are performed based on a multi-scale hierarchical decomposition graph to obtain multiple groups of preliminary action classification results of the human skeleton data.

[0075] The method for feature extraction and classification is as follows:

[0076] The method for obtaining the single-point hierarchical decomposition diagram is used to obtain the adjacency matrix of the human body hierarchical decomposition diagram. The initial joint points are the hip joint point, the abdomen joint point and the chest joint point, respectively, and three types of single-point hierarchical decomposition diagrams starting from the joint point of the center of gravity of the human body can be obtained, including the adjacency matrix of the single-point hierarchical decomposition diagram with the hip joint point [1] as the center of gravity joint point. The adjacency matrix of the single-point hierarchical decomposition graph with the abdominal joint point [2] as the center of gravity and the adjacency matrix of the single-point hierarchical decomposition graph with the chest joint

[21] as the center of gravity The detailed human body layered decomposition diagram corresponding to the single-point decomposition point set of each layer is shown in Table 1. The example of the single-point layered decomposition diagram with the chest joint point

[21] as the center of gravity joint point is shown in Figure 4 ,From the figure, we can observe that the hierarchical decomposition graph structures constructed from different starting points have different structures at the corresponding levels.,These different hierarchical decomposition graph structures can extract different features in the graph convolutional network,,making the feature set more comprehensive and rich.

[0077] Table 1 Three different single-point hierarchical decomposition graphs in the convolutional network of the human body feature map

[0078]

[0079] The adjacency matrix of the single-point hierarchical decomposition graph with the hip joint [1] as the center of gravity The adjacency matrix of the single-point hierarchical decomposition graph with the abdominal joint point [2] as the center of gravity and the adjacency matrix of the single-point hierarchical decomposition graph with the chest joint

[21] as the center of gravity Nine layers of graph convolution operations are performed respectively, and then three groups of preliminary action classification results of the overall hierarchical decomposition graph of the human body are obtained by the fully connected layer.

[0080] The method for obtaining the double-point hierarchical decomposition graph is used to obtain the adjacency matrix of the local hierarchical decomposition graph of the human body. 1 and the initial joint point n 2 By using the left and right shoulder joints, the left and right elbow joints, the left and right hip joints and the left and right knee joints respectively, four kinds of double-point hierarchical decomposition graphs starting from the symmetrical points of the limbs can be obtained, including the adjacency matrix of the double-point hierarchical decomposition graph starting from the left and right shoulder joints [5,9] Adjacency matrix of the two-point hierarchical decomposition graph starting from the left and right elbow joints [6,10] The adjacency matrix of the two-point hierarchical decomposition graph starting from the left and right hip joints [13,17] and the adjacency matrix of the two-point hierarchical decomposition graph starting from the left and right knee joints [14,18] The double-point decomposition point sets of each layer corresponding to the detailed human body local layered decomposition diagram are shown in Table 2. The double-point layered decomposition diagram examples starting from the left and right shoulder joints [5,9] are shown in Figure 4 .

[0081] Table 2 Four different double-point hierarchical decomposition graphs in the convolutional network of the local feature map of the human body

[0082]

[0083] The adjacency matrix of the double-point hierarchical decomposition graph starting from the left and right shoulder joints [5,9] Adjacency matrix of the two-point hierarchical decomposition graph starting from the left and right elbow joints [6,10] The adjacency matrix of the two-point hierarchical decomposition graph starting from the left and right hip joints [13,17] and the adjacency matrix of the two-point hierarchical decomposition graph starting from the left and right knee joints [14,18] Nine layers of graph convolution operations are performed respectively, and then four groups of preliminary action classification results of the local hierarchical decomposition graph of the human body are obtained by the fully connected layer.

[0084] The method for obtaining the single-point hierarchical decomposition diagram is used to obtain the adjacency matrix of the human body detail hierarchical decomposition diagram. The single-point hierarchical decomposition diagram, which adds the hand and upper limb-head connection hierarchical graph structure on the basis of the human body overall hierarchical decomposition diagram, includes three types, namely, the adjacency matrix of the detail single-point hierarchical decomposition diagram with the hip joint point as the center of gravity joint point Adjacency matrix of the detailed single-point hierarchical decomposition graph with the abdominal joint as the center of gravity And the adjacency matrix of the detailed single-point hierarchical decomposition graph with the chest joint as the center of gravity The hand and upper limb-head connection hierarchical graph structure is H 1 ={23,22,24,25},H 2 ={8,12,6,10,3,4}, with the chest joint point

[21] as the center of gravity, and the detailed single-point hierarchical decomposition diagram of the joint point, for example Figure 4 shown.

[0085] Adjacency matrix of the detailed single-point hierarchical decomposition graph with the hip joint as the center of gravity Adjacency matrix of the detailed single-point hierarchical decomposition graph with the abdominal joint as the center of gravity And the adjacency matrix of the detailed single-point hierarchical decomposition graph with the chest joint as the center of gravity Nine layers of graph convolution operations are performed respectively, and then three groups of preliminary action classification results of the human body detail hierarchical decomposition graph are obtained by the fully connected layer.

[0086] Since there are two forms of human skeleton data, namely bone point form and skeleton form, a total of 20 groups of preliminary action classification results can be obtained.

[0087] The graph convolution operation is:

[0088] The adjacency matrix of the hierarchical decomposition graph is subjected to graph convolution operations to capture the overall, local and fine motion features of the human body; then the attention-guided hierarchical aggregation operation is performed on the results of the graph convolution operation to assign weights to the hierarchical structure in the hierarchical decomposition graph, focusing on the most important hierarchical structure; the graph convolution operation and hierarchical aggregation operation jointly extract the spatial information of the human skeleton sequence data; finally, the temporal information of the human skeleton sequence data is extracted through the temporal convolution operation. After 9 layers of graph convolution operations, the accuracy of feature extraction in the multi-scale hierarchical decomposition graph is greatly improved.

[0089] The calculation process of the graph convolution operation is:

[0090]

[0091] Among them, F in and F outRepresents the input and output of the graph convolution operation; Θ represents the point-by-point convolution operation; the normalized adjacency matrix Λ is the degree matrix of the adjacency matrix A.

[0092] The multiple groups of preliminary action classification results of the human skeleton data are the scores of all given types of actions that the human skeleton data may be classified into, expressed as S = 20 represents the number of streams, N represents the number of action samples corresponding to the human skeleton data, I = 60 represents the number of categories of the given type of action, and each Indicates the score of the action samples corresponding to N human skeleton data being classified into I given types of actions.

[0093] S2: Perform a fusion operation on multiple groups of preliminary action classification results of the human skeleton data to obtain a group of action classification results of the human skeleton data.

[0094] The fusion operation is specifically as follows:

[0095] In this embodiment, the multi-classification problem can be regarded as 60 multivariate regression problems, and a regression model is defined for each type of action:

[0096] P i =f(r 1 , r 2 , ..., r i , ..., r S )

[0097] Among them, r i is a set of preliminary action classification results, P i It indicates the score of the action sample belonging to the i-th action, and the predicted score of the j-th action sample on the i-th action is P ij ∈P i ;

[0098] Then the multi-classification problem is transformed into a multivariate regression problem. The regression model is used to perform multivariate regression calculations on multiple groups of preliminary action classification results of human skeleton data to obtain the prediction score of each action sample. The maximum value is judged based on the prediction score of each action sample: if it satisfies That is, the prediction score P for the jth action sample on the I′th action I′j Not less than the predicted score P in any other category ij , then the determined action type of the jth action sample is the I′th action.

[0099] Compared with the simple averaging method and weighted averaging method that require artificially given weight values, the function f(·) here is obtained based on data, which can not only reduce human intervention, but also obtain different fusion functions for different action types, which can more accurately reflect the importance of each flow to the current action type, thereby improving the recognition accuracy.

[0100] Embodiment 2:

[0101] A multi-scale graph convolutional network adaptive integration system for human skeleton action recognition, using the multi-scale graph convolutional network adaptive integration method for human skeleton action recognition described in Example 1, such as Figure 2 As shown, it includes a sequentially connected input network, a multi-scale graph convolutional network, and an adaptive integration network;

[0102] The input network is used to obtain human skeleton data and construct a multi-scale hierarchical decomposition graph based on the human skeleton data;

[0103] The multi-scale graph convolutional network extracts features from the human skeleton data and classifies them based on the multi-scale hierarchical decomposition graph to obtain multiple groups of preliminary action classification results for the human skeleton data;

[0104] The multi-scale graph convolution network includes a parallel human body overall feature graph convolution network, a human body local feature graph convolution network and a human body detail feature graph convolution network;

[0105] The human body overall feature graph convolutional network is used to obtain the human body overall hierarchical decomposition graph, perform multi-level feature extraction and classification based on the human body overall hierarchical decomposition graph, and extract the overall movement trend of the human body; the human body overall feature graph convolutional network is composed of three parallel graph convolutional networks, and these three networks have different starting points in the hierarchical decomposition graph structure;

[0106] The human body local feature map convolution network is used to obtain the human body local hierarchical decomposition map, perform multi-level feature extraction and classification based on the human body local hierarchical decomposition map, and extract the movement status of the human body local structure; the human body local feature map convolution network is composed of 4 parallel graph convolution networks, and these 4 networks have different starting points in the hierarchical decomposition map structure;

[0107] The human body detail feature graph convolutional network is used to obtain a hierarchical decomposition graph of human body details, perform multi-level feature extraction and classification based on the hierarchical decomposition graph of human body details, and capture fine movement features; the human body detail feature graph convolutional network is composed of three parallel graph convolutional networks, and these three networks have different starting points in the hierarchical decomposition graph structure.

[0108] The graph convolution network is used to extract features and classify multi-scale hierarchical decomposition graphs respectively, and each graph convolution network corresponds to a stream. The graph convolution network includes 9 identical graph convolution operation layers connected in sequence and a fully connected layer; the graph convolution operation layer includes a graph convolution operation module, a hierarchical aggregation module and a temporal convolution module connected in sequence. The above 10 graph convolution networks use different hierarchical decomposition graph structures for feature extraction. The graph convolution network structure is as follows Figure 5 shown.

[0109] Since the human skeleton data has two forms, namely bone point form and skeleton form, the multi-scale graph convolutional network has a total of 20 streams, corresponding to 20 groups of preliminary action classification results, among which the human body overall feature graph convolutional network contains 6 streams, the human body local feature graph convolutional network contains 8 streams, and the human body detail feature graph convolutional network contains 6 streams.

[0110] The adaptive ensemble network is used to fuse the preliminary motion classification results of multiple groups of human skeleton data to obtain a group of motion classification results of human skeleton data.

[0111] The regression model in the adaptive ensemble network is represented by a neural network, such as Figure 6 As shown;

[0112] The neural network used in the adaptive ensemble network includes an input layer, a hidden layer, and an output layer connected in sequence.

[0113] The input layer is used to receive multiple groups of preliminary action classification results of human skeleton data output by the multi-scale graph convolutional network;

[0114] The hidden layer is used to perform multivariate regression calculation on multiple groups of preliminary action classification results of human skeleton data to obtain the predicted probability of each action sample; the hidden layer includes a fully connected layer I, a ReLU activation function layer and a fully connected layer II connected once;

[0115] The output layer is used to determine the maximum value based on the predicted probability of each action sample, and output the action sample and its corresponding determined action type.

[0116] The adaptive ensemble network transforms the classification problem into a multivariate regression problem, and the network loss function uses the mean square error loss. The adaptive ensemble network adopts the Adam optimizer during the training process, and the learning rate and weight decay are 0.001 and 1e-5 respectively, and the learning rate is multiplied by 0.2 every 10 training cycles, and a total of 50 training cycles are performed.

[0117] Finally, the classification accuracy of the multi-scale graph convolutional network adaptive integration method for human skeleton action recognition proposed in this invention on the human skeleton action data in the NTU-RGB+D 60 dataset is shown in Table 3. Table 3 shows the performance of the multi-scale graph convolutional network adaptive integration method for human skeleton action recognition on the NTU-RGB+D 60 dataset. NTU-RGB+D60 is a large dataset for skeleton action recognition. It contains 56,880 skeleton action samples performed by 40 different participants and divided into 60 categories. The authors of this dataset recommend two benchmarks. (1) Cross-subject (X-Sub): 20 of the 40 subjects' actions are used for training, and the remaining 20 are used for validation. (2) Cross-view (X-View): Two of the three camera views are used for training, and the other is used for validation.

[0118] Table 4

[0119]

[0120] Note: The ones marked with “” are baseline model performances; the ones in bold are the highest values ​​of quantitative indicators.

[0121] In order to verify the effectiveness of the multi-scale graph convolution network, Table 3 shows the classification results of the human body overall, local, and detail feature graph convolution network and the multi-scale graph convolution network respectively. It can be found that no matter whether the simple average integration method or the adaptive integration method is used, the classification accuracy of the human body overall, local, and detail feature graph convolution network is not much different, and the classification accuracy of the multi-scale graph convolution network is significantly improved compared with them, indicating that the multi-scale graph convolution network is effective in improving the accuracy of human skeleton action recognition. In order to verify the effectiveness of the adaptive integration network, Table 3 shows the classification accuracy of each network using the simple average integration method (the integration method used in HD-GCN) and the adaptive integration method when integrating multiple stream networks. It can be seen that the classification accuracy of the network using the adaptive integration method is slightly higher than that using the simple average method, which reflects that the adaptive integration network has a certain effect on fusing multi-stream network features and improving the overall classification accuracy of the model. In summary, the multi-scale graph convolution network adaptive integration method for human skeleton action recognition proposed in the present invention is effective in improving the accuracy of human skeleton action recognition.

[0122] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. A multi-scale graph convolutional network adaptive integration method for human skeleton action recognition, characterized in that: Includes steps: S0: Obtain human skeleton data and construct a multi-scale hierarchical decomposition graph based on the human skeleton data; S1: Extract features from human skeleton data and classify them based on a multi-scale hierarchical decomposition graph to obtain multiple groups of preliminary action classification results for human skeleton data; S2: Perform a fusion operation on multiple groups of preliminary action classification results of the human skeleton data to obtain a group of action classification results of the human skeleton data.

2. The multi-scale graph convolutional network adaptive integration method for human skeleton action recognition according to claim 1, characterized in that: The human skeleton data includes a bone point form and a skeleton form; The multi-scale hierarchical decomposition diagram includes a human body overall hierarchical decomposition diagram, a human body local hierarchical decomposition diagram and a human body detail hierarchical decomposition diagram; The human body overall layered decomposition diagram is a single-point layered decomposition diagram starting from the joint point of the human body's center of gravity; The human body local layered decomposition diagram is a double-point layered decomposition diagram starting from the symmetrical points of the limbs; The human body detail layered decomposition diagram is a single-point layered decomposition diagram to which the hand and upper limb-head connection diagram structure is added based on the human body overall layered decomposition diagram.

3. The multi-scale graph convolutional network adaptive integration method for human skeleton action recognition according to claim 2, characterized in that: The method for obtaining the double-point hierarchical decomposition diagram is as follows: select the initial joint point n1 and the initial joint point n2 of the symmetrical relationship, and start from the initial joint point n1 and the initial joint point n2 respectively, and perform single-point hierarchical decomposition on the human skeleton to obtain two single-point decomposition point set sets. and Then merge the two single point decomposition point sets and Get the set of double point decomposition points Single point decomposition point set Single point decomposition point set and double point decomposition point set Contains N L point sets; finally, the point set is decomposed according to the double point The adjacency matrix of the double-point hierarchical decomposition graph is constructed using the method of constructing the single-point hierarchical decomposition graph.

4. The multi-scale graph convolutional network adaptive integration method for human skeleton action recognition according to claim 3, characterized in that: The human body center of gravity joints include hip joints, abdomen joints and chest joints; The symmetrical points of the limbs include left and right shoulder joints, left and right elbow joints, left and right hip joints, and left and right knee joints; The hand and upper limb-head connection graph structure includes a finger joint point set and an upper limb-head joint point set, wherein the upper limb-head joint point set includes a wrist joint, an elbow joint and a point representing the head.

5. The multi-scale graph convolutional network adaptive integration method for human skeleton action recognition according to claim 3 or 4, characterized in that: The method for extracting features from human skeleton data and classifying them based on a multi-scale hierarchical decomposition graph is as follows: an adjacency matrix of the human body overall hierarchical decomposition graph and an adjacency matrix of the human body detail hierarchical decomposition graph are obtained from the human body overall hierarchical decomposition graph and the human body detail hierarchical decomposition graph respectively, and an adjacency matrix of the human body local hierarchical decomposition graph is obtained from the human body local hierarchical decomposition graph using the method for obtaining the double-point hierarchical decomposition graph; then multi-layer graph convolution operations are performed on the adjacency matrix of the human body overall hierarchical decomposition graph, the adjacency matrix of the human body detail hierarchical decomposition graph, and the adjacency matrix of the human body local hierarchical decomposition graph respectively, and then a plurality of groups of preliminary action classification results corresponding to the human body overall hierarchical decomposition graph, a plurality of groups of preliminary action classification results corresponding to the human body detail hierarchical decomposition graph, and a plurality of groups of preliminary action classification results corresponding to the human body local hierarchical decomposition graph are obtained by a fully connected layer.

6. The multi-scale graph convolutional network adaptive integration method for human skeleton action recognition according to claim 5, characterized in that: The fusion operation in step S2 is specifically as follows: For each type of action, define a regression model: P i =f(r1,r2,…,r i ,…,r S ) Among them, r i is a set of preliminary action classification results, P i It indicates the score of the action sample belonging to the i-th action, and the predicted score of the j-th action sample on the i-th action is P ij ∈P i ; The regression model is used to perform multiple regression calculations on multiple groups of preliminary action classification results of human skeleton data to obtain the prediction score of each action sample; the maximum value is judged based on the prediction score of each action sample: if it satisfies P I′j ≥P ij , then the determined action type of the jth action sample is the I′th action.

7. A multi-scale graph convolutional network adaptive integration system for human skeleton action recognition, using the multi-scale graph convolutional network adaptive integration method for human skeleton action recognition according to claims 1 to 6, characterized in that: It includes sequentially connected input networks, multi-scale graph convolutional networks, and adaptive integrated networks; The input network is used to obtain human skeleton data and construct a multi-scale hierarchical decomposition graph based on the human skeleton data; The multi-scale graph convolutional network extracts features from the human skeleton data and classifies them based on the multi-scale hierarchical decomposition graph to obtain multiple groups of preliminary action classification results for the human skeleton data; The adaptive integrated network is used to perform a fusion operation on preliminary action classification results of multiple groups of human skeleton data to obtain an action classification result of a group of human skeleton data.

8. The multi-scale graph convolutional network adaptive integration system for human skeleton action recognition according to claim 7, characterized in that: The multi-scale graph convolution network includes a parallel human body overall feature graph convolution network, a human body local feature graph convolution network and a human body detail feature graph convolution network; The human body overall feature map convolutional network is used to obtain a human body overall hierarchical decomposition map, and multi-level feature extraction and classification are performed according to the human body overall hierarchical decomposition map; The human body local feature map convolution network is used to obtain a human body local hierarchical decomposition map, and perform multi-level feature extraction and classification according to the human body local hierarchical decomposition map; The human body detail feature map convolutional network is used to obtain a human body detail hierarchical decomposition map, and perform multi-level feature extraction and classification based on the human body detail hierarchical decomposition map.

9. The multi-scale graph convolutional network adaptive integration system for human skeleton action recognition according to claim 8, characterized in that: The human body overall feature graph convolution network includes 3 parallel graph convolution networks; the human body local feature graph convolution network includes 4 parallel graph convolution networks; the human body detail feature graph convolution network includes 3 parallel graph convolution networks; The graph convolutional network uses different hierarchical decomposition graph structures for feature extraction.

10. The multi-scale graph convolutional network adaptive integration system for human skeleton action recognition according to claim 9, characterized in that: The adaptive integrated network includes an input layer, a hidden layer and an output layer connected in sequence; The input layer is used to receive multiple groups of preliminary action classification results of human skeleton data output by the multi-scale graph convolutional network; The hidden layer is used to perform multivariate regression calculation on multiple groups of preliminary action classification results of human skeleton data to obtain the predicted probability of each action sample; the hidden layer includes a fully connected layer I, a ReLU activation function layer and a fully connected layer II connected in sequence; The output layer is used to determine the maximum value based on the predicted probability of each action sample, and output the action sample and its corresponding determined action type.

Images