Factory-based multi-modal hypergraph convolution double-person interaction behavior identification method

Abstract

The invention discloses a multi-modal hypergraph convolution double-person interaction behavior recognition method based on a factory area, and the method comprises the steps: S1, taking two persons with interaction actions in the factory area as a whole, and constructing a double-person hypergraph; s2, respectively constructing an individual internal relation matrix and an interaction external relation matrix for important interaction joint points between two persons; s3, combining the step S1 and the step S2 to form a double-person interaction hypergraph; s4, performing symmetric processing on multi-stream input data (joint positions, movement speeds and skeletons), and constructing a space-time partial attention mechanism (ST-Part Attention, STPA) for the Officient-GCN; s5, carrying out early fusion on the three multi-stream inputs, and deploying the three multi-stream inputs into an Officient-GCN of an improved attention mechanism in combination with a double-person interaction hypergraph; and S6, after a final output feature map is obtained, the double-person interaction actions are classified. According to the method, rich individual and interaction characteristics in double-person actions in a factory can be effectively extracted, the model complexity is remarkably reduced, and the recognition precision is improved.

Description

Technical Field

[0001] The present invention relates to the field of computer vision technology, and in particular to a method for identifying two-person interactive behaviors based on a multimodal hypergraph convolution in a factory. Background Art

[0002] The recognition of two-person interaction behavior in the factory is to identify and understand the interaction behavior between two people through the continuous development of computer vision and machine learning technology, so as to analyze the normal and abnormal behaviors in the surveillance video in the factory. It has broad application prospects in many fields such as human-computer interaction and early warning assistance. At present, the recognition method based on the combination of joint point data and graph convolutional network has made satisfactory progress. However, this type of method still has the problem that the characteristic information of two-person interaction in the factory is not fully represented, and the model is also very complex and cannot be applied in practice.

[0003] In order to better capture the topological structure and connection pattern between human nodes, Yan et al. first proposed a spatial temporal graph convolutional network (Spatial Temporal GCN, ST-GCN), which uses joints as graph nodes and the natural connection between human structure and time as graph edges; then constructs a spatial temporal graph to extract high-level features from the skeleton sequence, and finally uses the Softmax function to distinguish action categories. This method provides an effective solution for effectively modeling the spatial and temporal relationship of actions, but the graph structure in the method is pre-calibrated and cannot flexibly represent the structure of different action samples. In order to make full use of the structural hierarchical information in the spatial domain and further improve the recognition accuracy, Shi et al. proposed a two-stream adaptive graph convolutional network (Two-stream Adaptive GCN, 2s-AGCN), which uses joint data and skeleton data (bone length and bone direction) respectively, and then fuses the results of the two streams to obtain the final recognition result. In order to capture complex spatiotemporal features, Chen et al. proposed a multi-scale spatial temporal graph convolutional network (MST-GCN), which divides the feature map into multiple segments along the channel dimension. Each segment uses residual connections to connect graph convolution and temporal convolution to enhance the multi-scale representation of the model. In order to incorporate semantic information into joint-level and frame-level representations to extract more features, Zhang et al. proposed semantic-guided neural networks (SGN), which introduces the semantic information of joints (joint type and frame index) into the network to enhance the feature representation ability. In order to simplify the model and improve the reasoning speed, some lightweight networks have been gradually proposed, such as DG-STGCN, Efficient-GCN, MART-GCN and other models. In order to effectively model the spatial relationship between two people and effectively extract the interaction features of two people, Li et al. proposed a two-person graph convolutional network (2p-GCN), which uses the two-person graph structure and geometric representation strategy to describe the interaction between two people, improving the accuracy of two-person interaction behavior recognition.

[0004] In summary, a lot of research results have been achieved in the recognition of two-person interactive behaviors based on graph convolution, but it still has the following limitations. (1) The commonly used graph structure lacks flexibility and only reflects the relationship of physical connections. It is difficult to fully represent the correlation between long-distance nodes in complex factories and complex actions. (2) When classifying interactive actions, the modeling of the rich interactive relationship between two people is not perfect. (3) The data is mainly based on joint points, and other semantic information derived has not been fully utilized.

[0005] Therefore, it is very meaningful to provide a factory-based multimodal hypergraph convolutional two-person interaction behavior recognition method (MH-GCN). Summary of the invention

[0006] In view of this, the present invention provides a multimodal hypergraph convolution method for identifying two-person interactive behaviors based on a factory area, so as to more accurately extract the interaction dependency information between structures and frames of two-person joints, and improve the recognition accuracy of two-person interactive behaviors in complex factory areas.

[0007] The technical solution of the present invention to solve the above technical problems is as follows: a method for identifying two-person interactive behaviors based on a multimodal hypergraph convolution in a factory area, comprising:

[0008] S1: Consider the two people who interact in the factory as a whole, use the joint point data, and combine natural connections and hyperconnections to construct a two-person hypergraph;

[0009] S2: For the important interaction points of the two people, the internal and external relationship matrices of the individuals are constructed respectively, and the degree of interaction between the interaction points is evaluated by the distance between the joints;

[0010] S3: Combining the two-person hypergraph and the relationship matrix to construct a two-person interaction hypergraph;

[0011] S4: Symmetrically process the multi-stream input data and build a spatiotemporal partial attention mechanism for Efficient-GCN; the multi-stream input data includes: joint position, movement speed, and skeleton;

[0012] S5: Early fusion of the three multi-stream input data; combined with the two-person interaction hypergraph, the fused multi-stream input data is deployed into the Efficient-GCN with improved attention mechanism, and the feature map is output.

[0013] S6: The final output feature map is globally averaged into a feature vector, and then a fully connected layer is used to classify the two-person interaction behavior.

[0014] Preferably, the S1 regards the two people who have interactive actions in the factory as a whole, uses the joint point data, and combines natural connections and hyperconnections to construct a two-person hypergraph, including:

[0015] S11: Consider the two people who interact in the factory as a whole, use the joint point data to rearrange the order of the joint points; represent the skeleton sequence of the two people as a hypergraph topology with joints as vertices and bones as edges;

[0016] S12: Physically connect all joints of the two people and connect the limbs with hyperedges to achieve information exchange between joints;

[0017] S13: The degrees of all vertices constitute the vertex degree matrix, the degrees of all hyperedges constitute the hyperedge degree matrix, and the adjacency matrix of the hypergraph represents the connection relationship between the joints of two people.

[0018] Preferably, S4 processes the multi-stream input data symmetrically and constructs a spatiotemporal partial attention mechanism for Efficient-GCN, including:

[0019] S41: Exchange the joint labels of each sample in the multi-stream input data to create a symmetric sequence, and generate a symmetric graph for each sample with active and passive behavior exchange;

[0020] S42: Randomly sample subsequences from the two-person interaction action samples and adjust the training sample size to 64 frames;

[0021] S43: The body is divided into five parts including the head and limbs and the spatiotemporal attention is allocated to them respectively. That is, the final spatiotemporal partial attention map is obtained through the frame-level and part-level pooling layers, fully connected layers, cascades, and element-by-element product operations.

[0022] The present invention provides a method for identifying two-person interaction behaviors based on multimodal hypergraph convolution in a factory area. The method first constructs a two-person interaction hypergraph using hypergraph convolution and a two-person interaction relationship matrix, which can more accurately express the original information between naturally connected and unnaturally connected joints in complex factory areas and action changes, as well as the potential interactive information in two-person behaviors. Secondly, the two-person interaction hypergraph is deployed in an Efficient-GCN with early fused three-stream data (joint position, skeleton, and movement speed) as input, and the spatiotemporal partial attention mechanism is used to distinguish body parts closely related to the behavior. Finally, the two-person interaction actions are classified, which can reduce the model training parameters, significantly reduce its computational complexity, and be more suitable for actual factories. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] Figure 1 The overall architecture diagram of the multimodal hypergraph convolution two-person interactive behavior recognition method (MH-GCN) based on the factory area provided by the disclosed embodiment of the present invention;

[0024] Figure 2 A two-person hypergraph structure diagram provided for the disclosed embodiment of the present invention;

[0025] Figure 3 A two-person interaction relationship matrix structure diagram provided for the disclosed embodiment of the present invention;

[0026] Figure 4 A multimodal input data diagram provided for an embodiment disclosed in the present invention;

[0027] Figure 5 A symmetric processing diagram of input data provided by the disclosed embodiment of the present invention;

[0028] Figure 6 A spatiotemporal partial attention map provided for an embodiment of the present invention;

[0029] Figure 7 A graph convolutional network deployment diagram provided for the disclosed embodiment of the present invention;

[0030] Figure 8 A comparison diagram of the confusion matrices provided for the disclosed embodiments of the present invention. DETAILED DESCRIPTION

[0031] The principles and features of the present invention are described below in conjunction with the accompanying drawings. The examples given are only used to explain the present invention and are not used to limit the scope of the present invention.

[0032] The multimodal hypergraph convolutional two-person interaction behavior recognition method based on the factory area (MH-GCN) has the following overall network structure: Figure 1 Specifically, the method comprises the following steps:

[0033] Step 1: Consider the two people who interact in the factory as a whole, and construct a two-person hypergraph by combining natural connections and hyper connections.

[0034] Specifically,

[0035] 1) If Figure 2 As shown in . Using the joint point data, we regard the two people as a whole, rearrange the joint points in order, and make physical connections and hyperedge connections for all the related nodes, as shown in Figure 2 (a). The hypergraph can be viewed as an approximate graph learning problem, consisting of pairs of hyperedges connected by hypernodes. Figure 2 (b) shows. The two-person hypergraph is represented as G H =(V,E H ). Where V = {v i |i=1,2,…,2N} represents the set of all vertices in a frame, E H ={e ij |i,j=1,2,…,2N} represents the set of all hyperedges connecting vertices. i With v j If there is a hyperedge connection between them, it is 1, otherwise it is 0. The calculation formula is:

[0036]

[0037] 2) Define the degree of a hyperedge as d(e), and the degrees of all hyperedges form the hyperedge degree matrix D e , the size of the matrix is ​​the number of hyperedges, and the diagonal items represent the number of hyperedges connected to the vertex. The calculation formula is:

[0038]

[0039] 3) Using the adjacency matrix A of the hypergraph H ∈R 2N×2N Construct a two-person connection relationship. Compared with ordinary graphs, each edge is not limited to the connection relationship between two adjacent points, and can achieve long-distance weighted communication of limbs.

[0040] Step 2: For the important interaction points of the two people, construct the individual internal and interaction external relationship matrices respectively, and evaluate the degree of interaction between the interaction points by the distance between the joints, such as Figure 3 As shown. Use Represents the correlation of geometric joints. The calculation formula for the correlation between two-person interaction joints is:

[0041]

[0042] Step 3: Combine the two-person hypergraph and the two-person interaction relationship matrix to construct a two-person interaction hypergraph.

[0043] Step 4: Process the multi-stream input data (joint position, motion speed, skeleton) symmetrically and build a spatiotemporal partial attention mechanism for Efficient-GCN.

[0044] Specifically,

[0045] 1) In order to obtain more original features, the input joint sequence is refined into three parts. The first part is as follows Figure 4 (a) shows the joint position of each joint point relative to the central joint point; the second part is as follows Figure 4 (b) shows the movement speed of each joint point in different frames; the third part is as follows Figure 4 (c) shows the length and angle of the bones.

[0046] 2) Many actions in two-person interaction are asymmetric, with active and passive individuals. Symmetric processing is used for all input skeleton sequences, that is, for each sample, the joint labels are exchanged to create a symmetric sequence, that is, a symmetric graph with active and passive behavior exchange is generated for each sample, such as Figure 5 shown.

[0047] 3) The spatiotemporal partial attention mechanism focuses on the five parts of the head and limbs of two people. First, the spatiotemporal input feature maps are pooled separately, and then the pooled feature vectors are connected together and compressed through the fully connected layer. Then, two independent fully connected layers are used to obtain two sets of attention scores for the frame and partial dimensions respectively. Finally, the two sets of attention scores are multiplied and calculated to obtain the final spatiotemporal attention map. Figure 6 As shown. The input and output calculation formulas of the attention module are:

[0048]

[0049] Step 5: The three multi-stream input data of the input branch are early fused through a concatenated operation and then sent to the main branch; combined with the two-person interaction hypergraph and the spatiotemporal partial attention mechanism, they are deployed in the Efficient-GCN with the improved attention mechanism, as shown in Figure 7 shown.

[0050] Step 6: The final output feature map is globally averaged as a feature vector, and then the fully connected layer is used to classify the two-person interaction behavior. The classification results are as follows: Figure 8 shown.

[0051] Experiments have shown that the proposed method for identifying two-person interaction behaviors based on multimodal hypergraph convolution (MH-GCN) in the factory area can not only capture the adjacent or distant spatiotemporal information of two people in the factory area while reducing the number of model parameters, but also make better use of the rich interaction information between two people. Experiments show that the accuracy of interaction behavior recognition reaches 98.24%, and the loss function value is 0.0589.

[0052] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. A method for identifying two-person interaction behaviors based on multimodal hypergraph convolution in a factory, characterized in that: include: S1: Consider the two people who interact in the factory as a whole, use the joint point data, and combine natural connections and hyperconnections to construct a two-person hypergraph; S2: For the important interaction points of the two people, the internal and external relationship matrices of the individuals are constructed respectively, and the degree of interaction between the interaction points is evaluated by the distance between the joints; S3: Combining the two-person hypergraph and the relationship matrix to construct a two-person interaction hypergraph; S4: Symmetrically process the multi-stream input data and build a spatiotemporal partial attention mechanism for Efficient-GCN; the multi-stream input data includes: joint position, movement speed, and skeleton; S5: Early fusion of the three multi-stream input data; combined with the two-person interaction hypergraph, the fused multi-stream input data is deployed into the Efficient-GCN with improved attention mechanism, and the feature map is output. S6: The final output feature map is globally averaged into a feature vector, and then a fully connected layer is used to classify the two-person interaction behavior.

2. The method for identifying two-person interaction behaviors based on multimodal hypergraph convolution in a factory area according to claim 1 is characterized in that: S1 regards the two people who have interactive actions in the factory as a whole, uses the joint point data, combines natural connections and hyperconnections to construct a two-person hypergraph, including: S11: Consider the two people who interact in the factory as a whole, use the joint point data to rearrange the order of the joint points; represent the skeleton sequence of the two people as a hypergraph topology with joints as vertices and bones as edges; S12: Physically connect all joints of the two people and connect the limbs with hyperedges to achieve information exchange between joints; S13: The degrees of all vertices constitute the vertex degree matrix, the degrees of all hyperedges constitute the hyperedge degree matrix, and the adjacency matrix of the hypergraph represents the connection relationship between the joints of two people.

3. The method for identifying two-person interaction behaviors based on multimodal hypergraph convolution in a factory area according to claim 1 is characterized in that: The S4 processes the multi-stream input data symmetrically and builds a spatiotemporal partial attention mechanism for Efficient-GCN, including: S41: Exchange the joint labels of each sample in the multi-stream input data to create a symmetric sequence, and generate a symmetric graph for each sample with active and passive behavior exchange; S42: Randomly sample subsequences from the two-person interaction action samples and adjust the training sample size to 64 frames; S43: The body is divided into five parts including the head and limbs and the spatiotemporal attention is allocated to them respectively. That is, the final spatiotemporal partial attention map is obtained through the frame-level and part-level pooling layers, fully connected layers, cascades, and element-by-element product operations.

Images