Plant-based multimodal hypergraph convolutional dual human interaction behavior recognition method

Abstract

The application discloses a plant-based multi-modal hypergraph convolution two-person interaction behavior recognition method, comprising the following steps: S1: regarding two persons with interaction actions in a plant as a whole, and constructing a two-person hypergraph; S2: constructing an individual internal and interaction external relationship matrix respectively for important interaction nodes between the two persons; step S3: combining step S1 and step S2 to form a two-person interaction hypergraph; step S4: performing symmetric processing on multi-stream input data (joint position, motion speed, skeleton), and constructing a space-time part attention mechanism (STPA) for Efficient-GCN; step S5: early fusion is performed on the three multi-stream inputs, and the two-person interaction hypergraph is combined, and then the two-person interaction hypergraph is deployed into the Efficient-GCN with the improved attention mechanism; and step S6: after obtaining a final output feature map, two-person interaction actions are classified. The method can effectively extract rich individual and interaction features in two-person actions in a plant, significantly reduce the model complexity, and improve the recognition accuracy.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computer vision, and in particular to a multi-modal hypergraph convolution double-person interaction behavior recognition method based on a factory area. BACKGROUND

[0002] Double-person interaction behavior recognition in a factory area is to recognize and understand the interaction behavior between two persons through the development of computer vision and machine learning technology, so as to analyze the normal and abnormal behaviors in the monitoring video in the factory area. It has broad application prospects in many fields such as human-computer interaction and early warning assistance. At present, the recognition method based on the combination of key node data and graph convolution network has made satisfactory progress. However, this kind of method still has the problems that the double-person interaction feature information representation in the factory area is insufficient, and the model is very complex and cannot be applied in practice.

[0003] To better capture the topology and connection patterns between human nodes, Yan et al. first proposed a spatial-temporal graph convolutional network (ST-GCN), which takes the joint nodes as graph nodes and the natural connection between human structure and time as graph edges. Then, a spatial-temporal graph is constructed to extract high-level features from the skeleton sequence, and finally, a softmax function is used to distinguish the action categories. This method provides an effective solution for effectively modeling the spatial and temporal relationships of actions, but the graph structure in the method is pre-labeled and cannot flexibly represent the structure of different action samples. To fully utilize the hierarchical information in the spatial domain and further improve the recognition accuracy, Shi et al. proposed a two-stream adaptive graph convolutional network (2s-AGCN), which uses joint node data and skeleton data (skeleton length and skeleton direction) respectively, and then fuses the results of the two streams to obtain the final recognition result. To capture complex spatial-temporal features, Chen et al. proposed a multi-scale spatial-temporal graph convolutional network (MST-GCN), which divides the feature map into multiple segments along the channel dimension, and each segment uses residual connection to connect graph convolution and temporal convolution, enhancing the multi-scale representation of the model. To incorporate semantic information into joint-level and frame-level representations and extract more features, Zhang et al. proposed semantics-guided neural networks (SGN), which introduced joint semantic information (joint type and frame index) into the network to enhance feature representation capability. To simplify the model and improve inference speed, some lightweight networks have been gradually proposed, such as DG-STGCN, Efficient-GCN, MART-GCN, etc. To effectively model the spatial relationship between two people and effectively extract the interaction features of two people, Li et al. proposed a two-person graph convolutional network (2p-GCN), which uses a two-person graph structure and a geometric representation strategy to describe the interaction between two people, improving the accuracy of two-person interaction behavior recognition.

[0004] In summary, two-person interaction behavior recognition based on graph convolution has made a lot of research achievements, but still has the following limitations. (1) The commonly used graph structure lacks flexibility and only reflects the relationship of physical connection, making it difficult to fully represent the relevance between distant nodes in complex factories and complex actions. (2) When classifying interactive actions, the modeling of the rich interaction between two people is not perfect. (3) Based on joint node data, other semantic information derived from it has not been fully utilized.

[0005] Therefore, it is very meaningful to provide a plant-based multi-modal hypergraph convolution dual-person interaction behavior recognition method (MH-GCN). SUMMARY

[0006] Therefore, the plant-based multi-modal hypergraph convolution dual-person interaction behavior recognition method is provided to more accurately extract the interaction dependency information between structures and frames of dual-person nodes of interest, and improve the recognition accuracy of dual-person interaction behavior in a complex plant.

[0007] The technical scheme for solving the above technical problems is as follows: the plant-based multi-modal hypergraph convolution dual-person interaction behavior recognition method comprises:

[0008] S1: regarding the dual persons with interactive actions in the plant as a whole, constructing a dual-person hypergraph by using the node data in combination with natural connection and hyperconnection;

[0009] S2: constructing individual internal and interactive external relationship matrices for important interactive nodes of the dual persons, and evaluating the interaction degree between the interactive nodes by the distance between the joints;

[0010] S3: combining the dual-person hypergraph and the relationship matrix to construct a dual-person interaction hypergraph;

[0011] S4: performing symmetric processing on the multi-stream input data, and constructing a space-time partial attention mechanism for the Efficient-GCN; wherein the multi-stream input data comprises joint position, motion speed and skeleton;

[0012] S5: performing early fusion on the three multi-stream input data; and combining the dual-person interaction hypergraph, deploying the fused multi-stream input data into the Efficient-GCN with the improved attention mechanism, and outputting a feature map

[0013] S6: globally averaging the final output feature map into a feature vector, and then using a fully connected layer to classify the dual-person interaction behavior.

[0014] Preferably, the S1 regards the dual persons with interactive actions in the plant as a whole, constructs a dual-person hypergraph by using the node data in combination with natural connection and hyperconnection, and comprises:

[0015] S11: regarding the dual persons with interactive actions in the plant as a whole, re-arranging the node order by using the node data; and representing the dual-person skeleton sequence as a hypergraph topology with joints as vertices and skeletons as edges;

[0016] S12: physically connecting and connecting all the nodes of the dual persons with four-limb hyperedges to realize information intercommunication between the nodes;

[0017] S13: The degrees of all vertices constitute a vertex degree matrix, the degrees of all hyperedges constitute a hyperedge degree matrix, and the adjacency matrix of the hypergraph represents the connection relationship between the dual human joint nodes.

[0018] Preferably, the S4 performs symmetric processing on the multi-flow input data, and constructs the space-time part attention mechanism for the Efficient-GCN, including:

[0019] S41: Exchange the joint labels of each sample in the multi-flow input data with each other to create a symmetric sequence, and generate a symmetric graph for each sample with active and passive exchange of behaviors;

[0020] S42: Randomly sample a sub-sequence from the dual human interaction action samples, and adjust the training sample size to 64 frames;

[0021] S43: Divide the body into five parts of head and four limbs and respectively assign space-time attention, that is, obtain the final space-time part attention graph through frame level, part level pooling layer, full connection layer, cascading and element-wise multiplication operation.

[0022] The application provides a multi-modal hypergraph convolution dual human interaction behavior recognition method based on a factory area. The method first constructs a dual human interaction hypergraph by using hypergraph convolution and a dual human interaction relationship matrix, which can more accurately express the original information between the natural connection and non-natural connection joint nodes in the complex factory area and the change of action, and the potential interaction information in the dual human behavior. Secondly, the dual human interaction hypergraph is deployed into the Efficient-GCN with early fusion of three flow data (joint position, skeleton and motion speed) as input, and a space-time part attention mechanism is used to distinguish the body parts closely related to the behavior. Finally, the dual human interaction action classification is performed, which can make the model training parameters less, significantly reduce the calculation complexity, and be more suitable for the actual factory area. BRIEF DESCRIPTION OF DRAWINGS

[0023] Figure 1 The multi-modal hypergraph convolution dual human interaction behavior recognition method (MH-GCN) based on a factory area provided by the disclosed embodiment of the application has the overall architecture diagram shown in the figure;

[0024] Figure 2 The dual human hypergraph structure diagram provided by the disclosed embodiment of the application;

[0025] Figure 3 The dual human interaction relationship matrix structure diagram provided by the disclosed embodiment of the application;

[0026] Figure 4 The multi-modal input data diagram provided by the disclosed embodiment of the application;

[0027] Figure 5 The symmetric processing diagram of the input data provided by the disclosed embodiment of the application;

[0028] Figure 6 A spatio-temporal partial attention diagram provided for the disclosed embodiments of the present application;

[0029] Figure 7 A graph convolutional network deployment diagram provided for the disclosed embodiments of the present application;

[0030] Figure 8 A confusion matrix comparison diagram provided for the disclosed embodiments of the present application. DETAILED DESCRIPTION

[0031] The principles and features of the present application are described below in conjunction with the accompanying drawings, which are only used to explain the present application and are not intended to limit the scope of the present application.

[0032] A multi-modal hypergraph convolutional two-person interaction behavior recognition method (MH-GCN) based on a factory site is shown in a whole network structure diagram as Figure 1 Specifically, the following steps are included:

[0033] Step 1: Two persons with interactive actions in a factory site are regarded as a whole, and a two-person hypergraph is constructed by combining natural connection and hyperconnection.

[0034] Specifically,

[0035] 1) as shown in Figure 2 . Using the node data, two people are regarded as a whole, and the node order is rearranged, and physical connection and hyperedge connection are performed on all nodes, as shown in Figure 2 (a). The hypergraph can be regarded as an approximate graph learning problem, which is composed of pairs of hyperedges connected by hypernodes, as shown in Figure 2 (b). The two-person hypergraph is represented as G H =(V,E H ). Where V={v i |i=1,2,…,2N} represents the set of all vertices in a frame, E H ={e ij |i,j=1,2,…,2N} represents the set of all hyperedges connected by vertices. If there is a hyperedge connection between vertices v i and v j , it is 1, otherwise it is 0. Its calculation formula is:

[0036]

[0037] 2) Define the degree of the hyperedge as d(e), and the degrees of all hyperedges constitute the hyperedge degree matrix D e , which has the size of the number of hyperedges, and the diagonal items represent the number of hyperedges connected to the vertices. Its calculation formula is:

[0038]

[0039] 3) Using the adjacency matrix A of the hypergraph H ∈R 2N×2N Build a two-person connection relationship. Compared with ordinary graphs, each edge is not limited to the connection between the two adjacent points, and long-distance four-limb weight interaction can be realized.

[0040] Step 2: For important interaction nodes of two people, respectively construct individual internal and interaction external relationship matrix, and evaluate the interaction degree between interaction nodes through the distance between joints, as shown in Figure 3 Use to represent the correlation of geometric joints, and the calculation formula of the correlation between two-person interaction joints is:

[0041]

[0042] Step 3: Combine the two-person hypergraph and the two-person interaction relationship matrix to construct a two-person interaction hypergraph.

[0043] Step 4: Symmetrically process the multi-stream input data (joint position, motion speed, skeleton), and construct the space-time part attention mechanism for Efficient-GCN.

[0044] Specifically,

[0045] 1) In order to obtain more original features, the input joint sequence is refined into 3 parts. The first part is shown in Figure 4 (a), which represents the joint position of each joint node relative to the center joint node; the second part is shown in Figure 4 (b), which represents the motion speed of each joint node in different frames; the third part is shown in Figure 4 (c), which represents the length and angle of the skeleton.

[0046] 2) Many actions in two-person interaction actions are asymmetric, and there are active individuals and passive individuals. Symmetric processing is adopted for all input skeleton sequences, that is, a symmetric sequence is created by exchanging the joint labels of each other for each sample, that is, an active and passive exchange symmetric graph is generated for each sample, as shown in Figure 5 .

[0047] 3) The space-time part attention mechanism focuses on the five parts of the heads and limbs of the two people. First, respectively pool the space-time input feature maps, then connect the collected feature vectors together and compress the information through the fully connected layer, then use 2 independent fully connected layers to respectively obtain 2 groups of attention scores in the frame and part dimensions, and finally multiply the 2 groups of attention scores and calculate the final space-time attention map. As shown in Figure 6 . The input and output calculation formulas of the attention module are respectively:

[0048]

[0049] Step 5: The 3 multi-flow input data of the input branch is early fused through a series operation, and then sent to the main branch; and combined with the double-person interaction hypergraph and the space-time partial attention mechanism, it is deployed into the Efficient-GCN with improved attention mechanism, as shown in Figure 7

[0050] Step 6: The obtained final output feature map is globally averaged as a feature vector, and then a fully connected layer is used to classify the double-person interaction behavior, and the classification result is as shown in Figure 8

[0051] Experiments prove that the multi-modal hypergraph convolution double-person interaction behavior recognition method (MH-GCN) based on the plant area can not only capture the adjacent or long-distance space-time information of the individuals inside the plant area, but also better utilize the rich interaction information between the double persons while reducing the model parameter amount. The experiment shows that the interaction behavior recognition accuracy reaches 98.24%, and the loss function value is 0.0589.

[0052] The above only describes the preferred embodiments of the present application and is not intended to limit the present application. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application shall be included in the protection scope of the present application.​​

Claims

1. A method for recognizing two-person interaction behavior based on multimodal hypergraph convolution in a factory area, characterized in that, include: S1: Treat two people with interactive actions within the factory area as a whole, and construct a two-person hypergraph using key data and combining natural connections and hyperconnections; S2: For key interaction points between two people, construct internal and external relationship matrices for each individual, and evaluate the degree of interaction between interaction points by the distance between the joints; S3: Combine the two-person hypergraph with the relationship matrix to construct a two-person interactive hypergraph; S4: Symmetrically process the multi-stream input data and construct a spatiotemporal partial attention mechanism for Efficient-GCN; the multi-stream input data includes: joint position, motion velocity, and skeleton; S5: Perform early fusion of the three multi-stream input data; and combine it with a two-person interactive hypergraph to deploy the fused multi-stream input data into Efficient-GCN with an improved attention mechanism, outputting feature maps. S6: The final output feature map is globally averaged into a feature vector, and then a fully connected layer is used to classify the two-person interaction behavior; S1 treats two people interacting within the factory area as a whole, and constructs a two-person hypergraph using keypoint data, combining natural connections and hyperconnections, including: S11: Treat the two people with interactive actions in the factory area as a whole, and rearrange the order of the joints using the joint data; represent the two-person skeleton sequence as a hypergraph topology with joints as vertices and bones as edges; S12: Physically connect all joints of the two characters and connect the four limbs via hyper-edge connections to achieve information exchange between joints; S13: The degree of all vertices forms the vertex degree matrix, the degree of all hyperedges forms the hyperedge degree matrix, and the adjacency matrix of the hypergraph represents the connection relationship between two key nodes. The S4 process the multi-stream input data symmetrically and constructs a spatiotemporal partial attention mechanism for Efficient-GCN, including: S41: Swap the joint labels of each sample in the multi-stream input data to create a symmetric sequence, generating a symmetric graph of active and passive behavior swaps for each sample; S42: Randomly sample subsequences from the two-person interaction action samples and adjust the training sample size to 64 frames; S43: Divide the body into five parts: head, limbs, etc., and assign spatiotemporal attention to each part. This is achieved by using frame-level and part-level pooling layers, fully connected layers, cascaded layers, and element-wise multiplication operations to obtain the final spatiotemporal partial attention map.

Images