A multi-view human behavior recognition method based on hypergraph learning
By constructing spatial and temporal hypergraphs and utilizing hypergraph neural networks to learn multi-view human behavior recognition methods, the problem of low accuracy in human behavior recognition in complex scenarios is solved, achieving more efficient behavior recognition results.
Patent Information
- Application Number
- CN202211440742.7
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2022-11-17
- Publication Date
- 2026-01-02
- Estimated Expiration
- 2042-11-17
AI Technical Summary
Existing human behavior recognition methods in complex scenarios cannot effectively model the complex correlations of key areas, resulting in low recognition accuracy, especially under conditions of lighting, occlusion, high dynamics, and changes in position and angle.
By constructing spatial and temporal hypergraphs, spatial dependencies and temporal correlations between multiple body skeletal points are captured through keypoint information. Hypergraph neural networks are then used for learning to achieve multi-view human behavior recognition.
It improves the accuracy of behavior recognition in complex scenes and solves problems such as object occlusion, insufficient lighting and weak key point correlation, and has the advantages of high efficiency and reliability.
Smart Images
Figure CN115830707B_ABST
Abstract
Description
TECHNICAL FIELD
[0001] The present application relates to the technical field of image processing, and particularly relates to a multi-view human behavior recognition method based on hypergraph learning. BACKGROUND
[0002] Behavior recognition is one of the representative tasks of computer vision, and accurate perception and recognition of human behaviors are important prerequisites for intelligent interaction and human-computer collaboration. In recent years, it has become a widely concerned research field, and has important significance for the research on body language interaction in application fields such as behavior analysis, intelligent driving, medical control, etc. With the continuous improvement of human joint point detection effect, it has been used for behavior recognition. However, the current method still has problems such as lack of time sequence modeling and lack of high-order semantic description of joint features.
[0003] In order to mine the time sequence relationship between multiple features in a video sequence, the traditional method uses a recurrent neural network to construct long-term association, which can pay attention to the information nodes in each frame by using a global context storage unit, and obtain more behavior features. Some other methods aim to use attention mechanism to aggregate the features of spatio-temporal image regions, effectively remove the influence of noise, etc., and improve the recognition accuracy. However, these methods still cannot effectively model the complex correlation of key regions, which is a major challenge for the behavior recognition task. Multi-view time sequence-based behavior recognition aims to use multi-view data, and models the time sequence information to better solve the information underdetermination problem caused by factors such as angle, illumination, occlusion, etc. in complex scenes, so as to enhance the feature information.
[0004] On March 18, 2014, a master's thesis entitled "Research on Human Behavior Recognition Algorithm Based on Hypergraph Clustering" was disclosed on Daoxie Baba, which aims to recognize human behaviors from videos containing human behaviors, and details the human behavior recognition method based on local features. A kind of human behavior recognition algorithm based on hypergraph clustering is proposed. First, the local features of human behaviors in videos and the methods for feature description are analyzed and compared, and efforts are made to study the construction of a dictionary about these features, to effectively represent the video with the words in the dictionary, and finally to use a classifier to classify the video, thereby completing the task of human behavior recognition. The disadvantage of this method is that in complex scenes, when encountering problems such as illumination, occlusion, high dynamics, position angle, etc., the accuracy of behavior recognition is low. SUMMARY
[0005] In order to solve the above technical problems, the application provides a multi-view human behavior recognition method based on hypergraph learning, which is directed to behaviors in a complex scene, and a spatial hypergraph is constructed by constructing multiple hypergraphs of human skeleton points in different views at the same time to capture the spatial dependency between multiple body skeleton points; a temporal hypergraph is constructed by constructing multiple hypergraphs of human skeleton points in different frames in the same view to better obtain the temporal correlation between features of specific joints in different views, so that learning is performed by a spatio-temporal hypergraph neural network according to the features constructed by the spatial hypergraph and the temporal hypergraph, and finally multi-view human behavior recognition based on hypergraph learning is realized.
[0006] The application provides a multi-view human behavior recognition method based on hypergraph learning, which comprises obtaining video data from P views, and further comprises the following steps:
[0007] Step 1: preprocessing the video data;
[0008] Step 2: constructing a spatial hypergraph according to joint information;
[0009] Step 3: constructing a temporal hypergraph according to joint information;
[0010] Step 4: performing feature learning on the spatial hypergraph and the temporal hypergraph by using a hypergraph neural network;
[0011] Step 5: extracting high-order information represented by the hypergraph to perform behavior recognition of human actions.
[0012] Preferably, the preprocessing method comprises segmenting the video data into N frames, extracting joint information of each frame by Openpose, storing the joint information in the form of joint coordinates x and y, saving the joint information as a json file, and constructing the spatial hypergraph and the temporal hypergraph according to the joint information.
[0013] In any of the above schemes, preferably, the spatial hypergraph refers to taking the joint as the vertex, dividing the human body into five parts of the trunk, the left hand, the right hand, the left leg and the right leg, connecting the joints of the same part in different views at the same time by a hyperedge, and constructing a spatial hypergraph according to the limb composition strategy The joint spatial information is aggregated, wherein, The vertex set of the spatial hypergraph is represented as V spa The hyperedge set is represented as W spa The weight of each hyperedge in the hyperedge set is represented as W, which is a weight matrix.
[0014] In any of the above schemes, preferably, the construction method of the spatial hypergraph comprises the following sub-steps:
[0015] Step 21: initializing the initial vertex feature of each spatial hypergraph as a feature matrix X neach row of the matrix is the coordinate of a human joint;
[0016] Step 22: generating the nth spatial hypergraph
[0017] Step 23: constructing the incidence matrix according to the vertex set and the hyperedge set;
[0018] Step 24: calculating the degree of the vertex in the nth spatial hypergraph and the degree of the vertex in the nth spatial hypergraph wherein, the function for calculating the degree of the vertex in the nth spatial hypergraph, the function for calculating the degree of the hyperedge in the nth spatial hypergraph, represents the ith joint in the nth frame in the pth view, represents the mth hyperedge in the nth spatial hypergraph;
[0019] Step 25: optimizing the network by using high-order information, the incidence matrix After Laplace transformation, the Laplace matrix is generated
[0020] In any of the above schemes, preferably, the nth spatial hypergraph The calculation formula is
[0021]
[0022] wherein, represents the vertex set of the nth spatial hypergraph, represents the hyperedge set of the nth spatial hypergraph, represents the weight of each hyperedge in the nth spatial hypergraph, n = 1, 2, …, N.
[0023] In any of the above schemes, preferably, the step 23 includes the incidence matrix of the nth spatial hypergraph represents the topological structure of the nth spatial hypergraph, if the vertex exists in a certain hyperedge, the corresponding element in the matrix is 1, otherwise 0.
[0024] In any of the above schemes, preferably, the incidence matrix of each spatial hypergraph is defined as:
[0025]
[0026] wherein, represents the ith joint in the nth frame in the pth view, represents the mth hyperedge in the nth spatial hypergraph, wherein m = 1, 2, …, M, M represents the number of hyperedges in a spatial hypergraph.
[0027] In any of the above solutions, preferably, the step 24 comprises calculating the degree of the vertices in the nth spatial hypergraph The calculation formula is:
[0028]
[0029] wherein, is the weight vector of the hyperedge
[0030] In any of the above solutions, preferably, the step 24 further comprises calculating the degree of the hyperedges in the nth spatial hypergraph The calculation formula is:
[0031]
[0032] and respectively represent the diagonal matrix of the hyperedge degree and the vertex degree in the nth spatial hypergraph.
[0033] In any of the above solutions, preferably, the Laplacian matrix The calculation formula is
[0034]
[0035] wherein, represents the inverse square root of the diagonal matrix of the vertex degree in the nth spatial hypergraph, represents the inverse of the diagonal matrix of the hyperedge degree in the nth spatial hypergraph.
[0036] In any of the above solutions, preferably, the temporal hypergraph refers to a hypergraph constructed by taking the joint nodes as vertices, dividing the sequence frames of the same view angle into a group, and connecting the same joint nodes of the same view angle sequence frame as a hyperedge wherein represents the vertex set of the temporal hypergraph, ε tem represents the hyperedge set thereof, W tem represents the weight of each hyperedge in the hyperedge set, and is a weight matrix.
[0037] In any of the above solutions, preferably, the construction method of the temporal hypergraph comprises the following sub-steps:
[0038] Step 31: initializing the initial vertex feature of each temporal hypergraph as a feature matrix X p , and each row of the matrix is the coordinate of the human joint node;
[0039] Step 32: generating multiple hypergraphs through P view angles
[0040] Step 33: constructing an incidence matrix according to the vertex set and the hyperedge set;
[0041] Step 34: calculating the degree of a vertex in the pth view time hypergraph and the degree of a hyperedge in the pth view time hypergraph
[0042] Step 35: optimizing the network using high-order information, the incidence matrix After Laplace transformation, a Laplace matrix is generated
[0043] In any of the above schemes, preferably, the step 33 includes the incidence matrix of the pth time hypergraph which represents the topological structure of the pth time hypergraph, if a vertex exists in a certain hyperedge, the corresponding element in the matrix is 1, otherwise 0.
[0044] In any of the above schemes, preferably, the incidence matrix of each time hypergraph is defined as:
[0045]
[0046] wherein, represents the qth hyperedge in the pth time hypergraph, q = 1, 2, …, Q, Q represents the number of hyperedges in a time hypergraph, and there are P time hypergraph incidence matrices in total.
[0047] In any of the above schemes, preferably, the degree of a vertex in the pth view time hypergraph is calculated by the formula:
[0048]
[0049] wherein, is the weight vector of the hyperedge .
[0050] In any of the above schemes, preferably, the degree of a hyperedge in the pth view time hypergraph is calculated by the formula:
[0051]
[0052] wherein, and respectively represent the diagonal matrix of the hyperedge degree and the vertex degree in the pth time hypergraph.
[0053] In any of the above schemes, preferably, the Laplace matrix is calculated by the formula:
[0054]
[0055] wherein, denotes the inverse square root of the diagonal matrix of vertex degrees in the p-th temporal hypergraph, denotes the inverse of the diagonal matrix of hyperedge degrees in the p-th temporal hypergraph.
[0056] In any of the above solutions, preferably, the hypergraph neural network comprises a spatial hypergraph neural network and a temporal hypergraph neural network.
[0057] In any of the above solutions, preferably, the spatial hypergraph neural network is composed of two spatial hypergraph basic blocks, each of which comprises two branches, each of which contains a 1x1 convolutional layer and a pooling layer.
[0058] In any of the above solutions, preferably, the construction method of the spatial hypergraph neural network comprises the following sub-steps:
[0059] Step 401: The feature matrices obtained by the two branches are spliced and trained by a multilayer perceptron (MLP);
[0060] Step 402: After aggregating the features by a 1x1 convolutional layer, the corresponding matrices are added element-wise, and a spatial hypergraph basic block adds the matrix Another hypergraph basic block adds the self-connection matrix I here;
[0061] Step 403: The feature matrices obtained by the two spatial hypergraph basic blocks are spliced to serve as the output of the spatial hypergraph neural network.
[0062] In any of the above solutions, preferably, the temporal hypergraph neural network is composed of 10 layers, a temporal hypergraph basic block 1 is used in the first layer, and a temporal hypergraph basic block 2 is used in the other layers, so as to realize effective learning and training of the time series feature information.
[0063] In any of the above solutions, preferably, the temporal hypergraph basic block 1 takes the vertex feature X as the input of five branches, each of which contains a 1x1 convolutional layer to reduce the channel dimension; the first and second branches respectively contain time convolutions with two different expansion rates, and extract feature information of different periods while reducing the number of parameters; the third and fifth branches respectively contain a 3x1 max-pooling layer for removing redundant information, and the results of the five branches are connected to obtain the output.
[0064] Preferably in any of the above solutions, the time hypergraph basic block 2 divides the vertex feature X into two parts x1 and x2, x1 is the input of the first four branches, and x2 is the input of the fifth branch, each branch contains the same network layer as the time hypergraph basic block 1.
[0065] Preferably in any of the above solutions, the step 5 comprises the following sub-steps:
[0066] Step 51: training the spatial hypergraph neural network to obtain spatial hypergraph features;
[0067] Step 52: training the time hypergraph neural network to obtain time hypergraph features;
[0068] Step 53: fusing the spatial hypergraph features and the time hypergraph features;
[0069] Step 54: calculating the behavior prediction probability value by Softmax;
[0070] Step 55: extracting the corresponding behavior action category with the maximum probability value as the prediction category.
[0071] Preferably in any of the above solutions, the step 51 comprises inputting an initialized feature matrix X n , a Laplacian matrix and a self-connection matrix I as the input of the spatial hypergraph neural network, f spatial is the output of the spatial hypergraph neural network, representing the spatial hypergraph features.
[0072] Preferably in any of the above solutions, an initialized feature matrix X p , a Laplacian matrix is inputted as the input of the time hypergraph neural network, wherein, only the fifth branch of the time hypergraph basic block is inputted, f temporal is the output of the time hypergraph neural network, representing the time hypergraph features.
[0073] The application provides a multi-view human body behavior recognition method based on hypergraph learning, which solves the problem of low behavior recognition accuracy caused by object occlusion, insufficient light, weak human body joint association and the like in a complex scene, and has the advantages of high efficiency and reliability. BRIEF DESCRIPTION OF DRAWINGS
[0074] Figure 1 is a flow chart of a preferred embodiment of the multi-view human body behavior recognition method based on hypergraph learning according to the application.
[0075] Figure 2Flow chart of another preferred embodiment of the multi-view human behavior recognition method based on hypergraph learning according to the present application.
[0076] Figure 3 Schematic diagram of an embodiment of the spatial hypergraph construction process of the multi-view human behavior recognition method based on hypergraph learning according to the present application.
[0077] Figure 4 Schematic diagram of an embodiment of the temporal hypergraph construction process of the multi-view human behavior recognition method based on hypergraph learning according to the present application.
[0078] Figure 5 Schematic diagram of an embodiment of the transformation process of the hypergraph and the association matrix of the multi-view human behavior recognition method based on hypergraph learning according to the present application.
[0079] Figure 6 Structural schematic diagram of an embodiment of the spatial hypergraph neural network of the multi-view human behavior recognition method based on hypergraph learning according to the present application.
[0080] Figure 7 Structural schematic diagram of an embodiment of the temporal hypergraph neural network of the multi-view human behavior recognition method based on hypergraph learning according to the present application. DETAILED DESCRIPTION
[0081] The present application will be further described below in conjunction with the accompanying drawings and specific embodiments.
[0082] Embodiment one
[0083] As shown in the figure, step 100 is performed to obtain video data from P views. Figure 1
[0084] Step 110 is performed to pre-process the video data, and the pre-processing method includes segmenting the video data into N frames, extracting the joint information of each frame by Openpose, storing the joint coordinates x and y, saving as a json file, and constructing a spatial hypergraph and a temporal hypergraph according to the joint information.
[0085] Step 120 is performed to construct a spatial hypergraph according to the joint information, and the spatial hypergraph refers to taking the joint as a vertex, dividing the human body into five parts of trunk, left hand, right hand, left leg and right leg, connecting the joints of the same part in different views at the same time with a hyperedge, and constructing a spatial hypergraph according to the limb composition strategy The joint spatial information aggregation is realized, wherein, denotes the vertex set of the spatial hypergraph, and ε spa denotes the hyperedge set thereof, and W spa is a weight matrix. The method for constructing the spatial hypergraph comprises the following sub-steps:
[0086] In step 121, the initial vertex feature of each spatial hypergraph is initialized as a feature matrix X n Each row of the matrix is the coordinate of a human body joint;
[0087] In step 122, the nthspatial hypergraph is generated The calculation formula is
[0088]
[0089] wherein, denotes the vertex set of the nthspatial hypergraph, denotes the hyperedge set of the nthspatial hypergraph, denotes the weight of each hyperedge in the nthspatial hypergraph, n = 1, 2, …, N.
[0090] In step 123, the association matrix is constructed according to the vertex set and the hyperedge set, and the association matrix of the nthspatial hypergraph is denotes the topological structure of the nthspatial hypergraph, if the vertex exists in a certain hyperedge, the corresponding element in the matrix is 1, otherwise it is 0, and the association matrix of each spatial hypergraph is defined as:
[0091]
[0092] wherein, denotes the ithjoint in the nthframe in the pthview, denotes the mthhyperedge in the nthspatial hypergraph, wherein m = 1, 2, …, M, and M represents the number of hyperedges in a spatial hypergraph.
[0093] In step 124, the degree of the vertex in the nthspatial hypergraph is calculated and the degree of the vertex in the nthspatial hypergraph is calculated The degree of the vertex in the nthspatial hypergraph is The calculation formula is:
[0094]
[0095] wherein, denotes a function for calculating the degree of the vertex in the nthspatial hypergraph, denotes a function for calculating the degree of the hyperedge in the nthspatial hypergraph, is the weight vector of the hyperedge .
[0096] The degree of the hyperedge in the nthspatial hypergraph is The calculation formula is:
[0097]
[0098] wherein, and respectively represent the diagonal matrix of the hyperedge degree and the vertex degree in the nth spatial hypergraph.
[0099] Step 125: optimizing the network by using high-order information, the association matrix The Laplacian matrix is generated by Laplace transformation The calculation formula is
[0100]
[0101] wherein, denotes the inverse square root of the diagonal matrix of the vertex degree in the nth spatial hypergraph, denotes the inverse of the diagonal matrix of the hyperedge degree in the nth spatial hypergraph.
[0102] Step 130 is performed to construct a time hypergraph according to the joint node information, wherein the time hypergraph refers to a hypergraph taking the joint node as a vertex, dividing the sequence frames of the same view angle into a group, and connecting the same joint nodes of the same view angle sequence frame by a hyperedge. wherein denotes the vertex set of the time hypergraph, ε tem denotes the hyperedge set thereof, W tem denotes the weight of each hyperedge in the hyperedge set, and is a weight matrix. The construction method of the time hypergraph comprises the following sub-steps:
[0103] Step 131 is performed to initialize the initial vertex feature of each time hypergraph as a feature matrix X p , wherein each row of the matrix is the coordinate of the human body joint node;
[0104] Step 132 is performed to generate a plurality of hypergraphs by P view angles wherein, denotes the pth time hypergraph, denotes the vertex set of the pth time hypergraph, denotes the hyperedge set of the pth time hypergraph, denotes the weight of each hyperedge in the pth time hypergraph;
[0105] Step 133 is performed to construct an association matrix according to the vertex set and the hyperedge set, wherein the association matrix of the pth time hypergraph denotes the topological structure of the pth time hypergraph, if the vertex exists in a certain hyperedge, the corresponding element in the matrix is 1, otherwise it is 0, and the association matrix of each time hypergraph is defined as:
[0106]
[0107] wherein, represents the qth hyperedge in the pth time hypergraph, q = 1, 2, …, Q, Q represents the number of hyperedges in a time hypergraph, and there are P time hypergraph association matrices in total.
[0108] Step 134 is performed to calculate the degree of a vertex in the pth view time hypergraph and the degree of a hyperedge in the pth view time hypergraph The degree of the vertex in the pth view time hypergraph is calculated. The calculation formula is:
[0109]
[0110] wherein, is the weight vector of the hyperedge .
[0111] The degree of the hyperedge in the pth view time hypergraph is calculated. The calculation formula is:
[0112]
[0113] wherein, and respectively represent the diagonal matrix of the hyperedge degree and the vertex degree in the pth time hypergraph.
[0114] Step 135 is performed to optimize the network using high-order information, and the association matrix is subjected to Laplace transformation to generate a Laplace matrix The calculation formula is:
[0115]
[0116] wherein, represents the inverse square root of the diagonal matrix composed of the vertex degree in the pth time hypergraph, represents the inverse of the diagonal matrix composed of the hyperedge degree in the pth time hypergraph.
[0117] Step 140 is performed to use a hypergraph neural network to learn features of the spatial hypergraph and the time hypergraph, and the hypergraph neural network includes a spatial hypergraph neural network and a time hypergraph neural network.
[0118] The spatial hypergraph neural network is composed of two spatial hypergraph basic blocks, each of which includes two branches, each of which contains a 1x1 convolutional layer and a pooling layer. The construction method of the spatial hypergraph neural network includes the following sub-steps:
[0119] In execution step 141, the feature matrices obtained by the two branches are spliced and trained through a multilayer perceptron (MLP).
[0120] In execution step 142, the features are aggregated through a 1x1 convolutional layer and then element-wise added to the corresponding matrix. The matrix added here by one spatial hypergraph basic block is the identity matrix I. The matrix added here by the other hypergraph basic block is the identity matrix I.
[0121] In execution step 143, the feature matrices obtained by the two spatial hypergraph basic blocks are spliced to serve as the output of the spatial hypergraph neural network.
[0122] The temporal hypergraph neural network is composed of 10 layers, in which temporal hypergraph basic block 1 is used in the first layer and temporal hypergraph basic block 2 is used in the other layers, thereby achieving effective learning and training of time series feature information. The temporal hypergraph basic block 1 takes vertex features X as input of five branches, each of which contains a 1x1 convolutional layer to reduce the channel dimension; the first and second branches contain time convolutions with two different expansion rates, respectively, and extract feature information of different periods while reducing the number of parameters; the third and fifth branches contain a 3x1 max-pooling layer, respectively, for removing redundant information, and the results of the five branches are connected to obtain the output. The temporal hypergraph basic block 2 divides the vertex features X into x1 and x2, x1 is input to the first four branches, and x2 is input to the fifth branch, and each branch contains the same network layers as the temporal hypergraph basic block 1.
[0123] In execution step 150, high-order information represented by the hypergraph is extracted for human action behavior recognition. The following sub-steps are included:
[0124] In execution step 151, the spatial hypergraph neural network is trained to obtain spatial hypergraph features, which are initialized as a feature matrix X n , a Laplacian matrix , and an identity matrix I as input of the spatial hypergraph neural network, f spatial , and the spatial hypergraph features as output of the spatial hypergraph neural network.
[0125] In execution step 152, the temporal hypergraph neural network is trained to obtain temporal hypergraph features, which are initialized as a feature matrix X p , a Laplacian matrix as input of the temporal hypergraph neural network, wherein, the fifth branch f only inputs the temporal hypergraph basic block, temporal as output of the temporal hypergraph neural network, representing the temporal hypergraph feature.
[0126] performing step 153, fusing the spatial hypergraph feature and the temporal hypergraph feature.
[0127] performing step 154, calculating the behavior prediction probability value by Softmax.
[0128] performing step 155, extracting the corresponding behavior action category with the maximum probability value as the prediction category.
[0129] Embodiment two
[0130] In order to realize accurate recognition of human behaviors in complex environments, such as Figure 2 As shown in the figure, the application provides a multi-view human behavior recognition method based on hypergraph learning, which recognizes video sequences of different views, uses hypergraphs to model human bodies in time and space, uses hypergraph neural networks to learn hypergraphs, and realizes human behavior recognition in complex environments.
[0131] 1. Video acquisition
[0132] Different cameras are used to acquire video data, and the multi-view video data is preprocessed. Video data is acquired from P views as input, and the video data is segmented into N frames. The joint information of each frame is extracted by Openpose, stored as joint coordinates x and y, saved as a json file, and the spatial hypergraph and the temporal hypergraph are constructed according to the joint information.
[0133] 2. Construction of spatial hypergraph
[0134] (1) For the spatial hypergraph, the joints are taken as the vertices, the human body is divided into five parts: torso, left hand, right hand, left leg and right leg, and the joints of the same part in different views at the same time are connected by a hyperedge, so as to construct a spatial hypergraph according to the limb composition strategy to realize the aggregation of spatial information of joints. Wherein, represents the vertex set of the spatial hypergraph, ε spa represents the hyperedge set, W spa represents the weight of each hyperedge in the hyperedge set, which is a weight matrix.
[0135] (2) The initial vertex feature of each spatial hypergraph is initialized as a feature matrix X n , and each row of the matrix is the coordinates of the human joint.
[0136] (3) Since N frames are extracted from each video sequence, multiple hypergraphs can be generated by N frames where, denotes the nth spatial hypergraph, denotes the vertex set of the nth spatial hypergraph, denotes the hyperedge set of the nth spatial hypergraph, denotes the weight of each hyperedge in the nth spatial hypergraph.
[0137] (4) Construct the incidence matrix according to the vertex set and the hyperedge set. The incidence matrix of the nth spatial hypergraph denotes the topological structure of the nth spatial hypergraph, if the vertex exists in a certain hyperedge, the corresponding element in the matrix is 1, otherwise it is 0. The incidence matrix of each spatial hypergraph is defined as:
[0138]
[0139] where, denotes the ith node of interest in the nth frame of the pth view, denotes the mth hyperedge in the nth spatial hypergraph, where m = 1, 2, …, M, M represents the number of hyperedges in a spatial hypergraph, n = 1, 2, …, N, there are N spatial hypergraph incidence matrices in total.
[0140] (5) The degree of the vertex in the nth spatial hypergraph The calculation formula is:
[0141]
[0142] where, is the weight vector of the hyperedge .
[0143] The degree of the hyperedge in the nth spatial hypergraph The calculation formula is:
[0144]
[0145] and respectively denote the diagonal matrix of the hyperedge degree and the vertex degree in the nth spatial hypergraph.
[0146] (6) In order to optimize the network by using high-order information, the incidence matrix is transformed by Laplace to generate the Laplace matrix The calculation formula is:
[0147]
[0148] 3. Construction of temporal hypergraph
[0149] (1) For the time hypergraph, the joint nodes are taken as the vertices, the same view sequence frames are divided into a group, and the same joint nodes of the same view sequence frames are connected by the hyperedge to construct a time hypergraph wherein represents the vertex set of the time hypergraph, ε tem represents the hyperedge set of the time hypergraph, W tem represents the weight of each hyperedge in the hyperedge set, and is a weight matrix.
[0150] (2) The initial vertex feature of each time hypergraph is initialized as the feature matrix W p , and each row of the matrix is the coordinate of the human joint node.
[0151] (3) Since there are P views, multiple hypergraphs can be generated through P views wherein represents the pth time hypergraph, represents the vertex set of the pth time hypergraph, represents the hyperedge set of the pth time hypergraph, represents the weight of each hyperedge in the pth time hypergraph.
[0152] (4) The association matrix is constructed according to the vertex set and the hyperedge set, and the association matrix of the pth time hypergraph represents the topological structure of the pth time hypergraph, if the vertex exists in a certain hyperedge, the corresponding element in the matrix is 1, otherwise it is 0. The association matrix of each time hypergraph is defined as:
[0153]
[0154] wherein, represents the qth hyperedge in the pth time hypergraph, q = 1, 2, …, Q, Q represents the number of hyperedges in a time hypergraph, and there are P time hypergraph association matrices in total.
[0155] (5) The degree of the vertex in the pth view time hypergraph is calculated according to the formula:
[0156]
[0157] wherein, is the weight vector of the hyperedge .
[0158] The degree of the hyperedge in the pth view time hypergraph is calculated according to the formula:
[0159]
[0160] Definitions and respectively represent the diagonal matrix of the hyperedge degree and vertex degree in the p-th time hypergraph.
[0161] (6) In order to optimize the network by using high-order information, the association matrix After Laplace transformation, the Laplace matrix is generated The calculation formula is:
[0162]
[0163] 4. Feature learning of hypergraph by hypergraph neural network
[0164] After the construction of the hypergraph, the spatial hypergraph neural network is used for feature learning of the spatial hypergraph, and the temporal hypergraph neural network is used for feature learning of the temporal hypergraph. The high-order information represented by the hypergraph is extracted, and the behavior recognition of human action is performed.
[0165] (1) Construction of spatial hypergraph neural network
[0166] For the spatial hypergraph neural network, it is composed of two spatial hypergraph basic blocks. Each spatial hypergraph basic block includes two branches, each branch contains a 1x1 convolution layer and a pooling layer, then the feature matrices obtained by the two branches are spliced, and are trained by a multilayer perceptron (MLP). After the features are aggregated by a 1x1 convolution layer, they are added element-wise to the corresponding matrix. The matrix added by one spatial hypergraph basic block here is The matrix added by the other hypergraph basic block here is the self-connection matrix I. Finally, the feature matrices obtained by the two spatial hypergraph basic blocks are spliced to serve as the output of the spatial hypergraph neural network.
[0167] (2) Construction of temporal hypergraph neural network
[0168] The time hypergraph neural network is composed of 10 layers, time hypergraph basic block 1 is used in the first layer, and time hypergraph basic block 2 is used in other layers, so as to realize effective learning and training of time sequence feature information. In order to carry out efficient learning and training and reduce the calculation amount in the network, the time hypergraph basic block 1 takes the vertex feature X as the input of five branches, each branch contains a 1x1 convolution layer to reduce the channel dimension; the first and second branches respectively contain time convolutions with two different expansion rates, and different period feature information is extracted while the number of parameters is reduced; the third and fifth branches respectively contain a 3x1 maximum pool layer for removing redundant information, and the results of the five branches are connected to obtain the output. The time hypergraph basic block 2 divides the vertex feature X into x1 and x2, x1 is input into the first four branches, and x2 is input into the fifth branch, and each branch contains the same network layer as the time hypergraph basic block 1.
[0169] (3) Training and prediction
[0170] The initialized feature matrix X n , Laplacian matrix L and self-connection matrix I are taken as inputs of the spatial hypergraph neural network, f spatial is the output of the spatial hypergraph neural network, representing the spatial hypergraph feature; the initialized feature matrix X p , Laplacian matrix L are taken as inputs of the time hypergraph neural network, wherein, only the fifth branch of the input time hypergraph basic block, f temporal is the output of the time hypergraph neural network, representing the time hypergraph feature. Finally, the obtained features are fused, and the behavior prediction probability value is calculated through Softmax, and finally the prediction category is the corresponding behavior action category with the maximum probability value.
[0171] Example three
[0172] The schematic diagram of the spatial hypergraph construction process is shown in Figure 3 . The present application takes all human body joint nodes in different perspectives at the same time as the vertex set of the hypergraph, connects the joint nodes of the same part in different perspectives at the same time with a hyperedge, and constructs a spatial hypergraph by taking the set of all hyperedges as the hyperedge set of the hypergraph. Since there are N frames in each perspective, a total of N spatial hypergraphs are constructed.
[0173] Example four
[0174] The schematic diagram of the time hypergraph construction process is shown in Figure 4The present application takes all human body joints in the same view at different time as the vertex set of the hypergraph, and the same joint in the same view at different time is connected by a hyperedge, and the set of all hyperedges constitutes the hyperedge set of the hypergraph, thereby constructing a time hypergraph. Since there are P views in total, P time hypergraphs are constructed in total.
[0175] Example Five
[0176] If the hypergraph is defined as wherein is the set of vertices in the hypergraph, and the elements in the set are denoted as v∈V. ε is the set of hyperedges in the hypergraph, and the elements in the set are denoted as e∈ε; W is the hyperedge weight matrix, recording the weight of each hyperedge, denoted as ω(e), and the relationship between the hyperedge and the vertex is represented by constructing the association matrix H, which is a matrix. Specifically, if the vertex v exists in the hyperedge e, then h(v, e) = 1, otherwise h(v, e) = 0, as shown in Figure 5 .
[0177] Example Six
[0178] As shown in Figure 6 , the spatial hypergraph neural network is composed of two spatial hypergraph basic blocks, each of which includes two branches, each of which contains a 1x1 convolution layer and a pooling layer, then the feature matrices obtained by the two branches are spliced, trained through a multilayer perceptron (MLP), and then aggregated through a 1x1 convolution layer, and the corresponding matrix is added element by element. The matrix added here by one spatial hypergraph basic block is The matrix added here by the other hypergraph basic block is the self-connection matrix I. Finally, the feature matrices obtained by the two spatial hypergraph basic blocks are spliced to serve as the output of the spatial hypergraph neural network
[0179] Example Seven
[0180] As shown in Figure 7As shown, the time hypergraph neural network is composed of 10 layers, time hypergraph basic block 1 is used in the first layer, and time hypergraph basic block 2 is used in other layers, so as to realize effective learning and training of time sequence feature information. In order to carry out efficient learning and training and reduce the calculation amount in the network, the time hypergraph basic block 1 takes the vertex feature X as the input of five branches, each branch contains a 1x1 convolution layer to reduce the channel dimension; the first and second branches contain time convolutions with two different expansion rates respectively, and different period feature information is extracted while the number of parameters is reduced; the third and fifth branches contain a 3x1 max pool layer respectively, which is used to remove redundant information, and the results of the five branches are connected to obtain the output. The time hypergraph basic block 2 divides the vertex feature X into x1 and x2, x1 is used as the input of the first four branches, and x2 is used as the input of the fifth branch, and each branch contains the same network layer as the time hypergraph basic block 1.
[0181] In order to better understand the present application, the above is described in detail in combination with the specific embodiments of the present application, but is not a limitation on the present application. Any simple modification made according to the technical essence of the present application to the above embodiments is still within the scope of the technical solution of the present application. In the specification, each embodiment focuses on the difference from other embodiments, and the same or similar parts of each embodiment can be referred to. For system embodiments, since they basically correspond to method embodiments, the description is relatively simple, and the relevant parts can be referred to the part of the method embodiment.
Claims
1. A multi-view human action recognition method based on hypergraph learning, comprising acquiring video data from P views, characterized in that, Further comprising the following steps: Step 1: preprocessing the video data, the method of preprocessing comprising segmenting the video data into N frames, extracting the joint information of each frame by Openpose, storing the joint coordinates x and y, saving as a json file, and constructing a spatial hypergraph and a temporal hypergraph according to the joint information; Step 2: Construct a spatial hypergraph according to the joint information, wherein the spatial hypergraph refers to taking the joint as a vertex, dividing the human body into five parts of a trunk, a left hand, a right hand, a left leg and a right leg, connecting the joints of the same part in different views at the same time with a hyperedge to construct a spatial hypergraph according to a limb composition strategy , and realizing joint spatial information aggregation, wherein denotes a vertex set of the spatial hypergraph, denotes a hyperedge set thereof, denotes a weight of each hyperedge in the hyperedge set, and is a weight matrix; Step 3: constructing a time hypergraph according to the joint information, the time hypergraph refers to taking the joint as a vertex, dividing the sequence frames of the same view into a group, and connecting the same joints of the sequence frames of the same view as a hyperedge to construct a hypergraph wherein a vertex set of the time hypergraph is represented by V, a hyperedge set thereof is represented by E, a weight of each hyperedge in the hyperedge set is represented by W, which is a weight matrix; Step 4: using a hypergraph neural network to learn the features of the spatial hypergraph and the temporal hypergraph; Step 5: extracting the high-order information represented by the hypergraph to recognize the behavior of the human body, comprising the following sub-steps: Step 51: training the spatial hypergraph neural network to obtain spatial hypergraph features; Step 52: training the temporal hypergraph neural network to obtain temporal hypergraph features; Step 53: fusing the spatial hypergraph features and the temporal hypergraph features; Step 54: calculating the behavior prediction probability value by Softmax; Step 55: extracting the corresponding behavior action category with the maximum probability value as the prediction category. 2.The method of claim 1, wherein, The method for constructing the spatial hypergraph comprises the following sub-steps: Step 21 : initializing the initial vertex features of each spatial hypergraph to a feature matrix each row of the matrix being the coordinates of a human body joint Step 22: Generate the nth spatial hypergraph ; Step 23: constructing an association matrix according to the vertex set and the hyperedge set; Step 24: Calculate the degree of the vertex in the nth spatial hypergraph and the degree in the nth spatial hypergraph , wherein, represents a function for calculating the degree of the vertex in the nth spatial hypergraph, represents a function for calculating the degree of the hyperedge in the nth spatial hypergraph, represents the ith joint node in the nth frame of the pth view, represents the mth hyperedge in the nth spatial hypergraph; Step 25: Optimizing the network with high-order information, association matrix The Laplacian matrix is generated by Laplace transformation . 3.The method of claim 2, wherein, The nth spatial hypergraph The calculation formula is ， wherein, denotes the vertex set of the nth spatial hypergraph, denotes the hyperedge set of the nth spatial hypergraph, denotes the weight of each hyperedge in the nth spatial hypergraph, n = 1, 2, …, N. 4.The method of claim 3, wherein, The step 23 includes the incidence matrix of the nth spatial hypergraph , which represents the topology of the nth spatial hypergraph. If a vertex exists in a certain hyperedge, the corresponding element in the matrix is 1, otherwise 0.
5. The multi-view human action recognition method based on hypergraph learning of claim 4, wherein, The association matrix of each spatial hypergraph is defined as: ， wherein, represents the ith joint node in the nth frame of the pth view, represents the mth hyperedge in the nth spatial hypergraph, where m = 1, 2, …, M, and M represents the number of hyperedges in a spatial hypergraph.
6. The multi-view human action recognition method based on hypergraph learning of claim 5, wherein, The step 24 includes calculating the degree of the vertex in the nth spatial hypergraph The calculation formula is ， wherein is a weight vector of the super-edge .
7. The multi-view human action recognition method based on hypergraph learning of claim 6, wherein, The step 24 further comprises calculating the degree of the hyperedge in the nth spatial hypergraph The calculation formula is: ， wherein and denote the diagonal matrices of the hyperedge degrees and vertex degrees in the nth spatial hypergraph, respectively.
8. The multi-view human action recognition method based on hypergraph learning of claim 7, wherein, The Laplacian matrix The formula for calculating is ， wherein, denotes the inverse square root of the diagonal matrix of vertex degrees in the nth spatial hypergraph, denotes the inverse of the diagonal matrix of hyperedge degrees in the nth spatial hypergraph.