A multi-group behavior recognition method based on space-time interaction reasoning

Abstract

The application provides a multi-group behavior recognition method based on space-time interaction reasoning, and mainly relates to how to effectively mine space-time interaction to perform individual behavior, group clustering and multi-group behavior recognition. The method proposes a space-time interaction relationship reasoning strategy, which focuses on the interaction relationship between individuals and groups, and considers appearance, position distance, space-time trajectory and other characteristics, so that the accuracy of behavior recognition is improved. First, the interaction relationship between individuals is explored by analyzing the characteristics of the nearest neighbors of individuals and the position distance characteristics. Second, the individuals are divided into multiple small groups by combining the behavior similarity and the space-time trajectory characteristics. Finally, a group interaction reasoning module is constructed to mine the interaction between different small groups and individuals, and individuals with less influence on group behavior are excluded according to the importance. The application fully considers the space-time interaction characteristics of individuals and groups, and solves the problem of insufficient mining of space-time interaction characteristics in behavior recognition.

Description

TECHNICAL FIELD

[0001] The present application relates to the individual behavior, group clustering and multi-group behavior recognition problem in the field of deep learning, and in particular to a multi-group behavior recognition method based on space-time interaction reasoning. BACKGROUND

[0002] Human activity understanding based on video is a core research field of computer vision, and has broad application prospects in the fields of monitoring systems and social scene understanding. Multi-group behavior recognition is a more complex field in human activity understanding research. Existing researches mostly extract individual features through convolutional neural networks, and then obtain group-level features through further reasoning and pooling operations. Unlike individual behavior recognition, multi-group behavior recognition pays more attention to the mining of interaction relationships. With the development of graph neural networks, related researches use graph convolution networks to infer the potential interaction relationships between individuals to enrich group features and improve the accuracy of behavior recognition.

[0003] However, the existing behavior recognition methods are not deep enough in the mining of space-time interaction relationships, resulting in inaccurate group-level features. Therefore, the present patent focuses on the mining of space-time interaction features, while considering appearance, location distance, space-time trajectory and other features, and uses a graph pooling network to give different degrees of attention to different individuals, thereby obtaining high-level group behavior features and improving the accuracy of behavior recognition. First, a 3D convolutional network is used to extract the space-time features of the video, and then an attention mechanism and a RoIAlign module are used to extract the features of each individual. Second, a Graph Attention Network and a Dynamic Inference Network are used to infer the interaction relationships between individuals and obtain the category of individual behavior. Then, the features of the nearest neighbors of the individuals and the location distance features are analyzed to explore the interaction relationships between individuals. Then, the individuals are divided into multiple small groups by combining the behavior similarity features and the space-time trajectory features. Finally, a group interaction reasoning module is constructed to mine the interactions between different small groups and individuals, and to exclude individuals who have little influence on group behavior according to their importance, and the multi-group behavior recognition result is obtained after inputting the classifier. SUMMARY

[0004] The purpose of the present application is to provide a multi-group behavior recognition method based on space-time interaction reasoning, which fully considers the space-time interaction features of individuals and groups, uses graph neural networks to deeply mine space-time interaction features, and considers appearance, location distance, space-time trajectory and other features, and uses a self-attention graph pooling network to give different degrees of attention to different individuals, thereby enriching the group-level features and solving the problem of insufficient mining of space-time interaction features in behavior recognition.

[0005] For the convenience of explanation, first introduce the following concepts:

[0006] Graph: a graph structure composed of nodes and edges connecting nodes.

[0007] Graph Convolutional Network (GCN): The purpose of GCN is to extract the spatial features of the topological graph, and the data processed is the graph structure, that is, the Non Euclidean Structure.

[0008] Graph Attention Network (GAT): GAT can flexibly learn the attention weight between nodes through parameterized operation based on self-attention strategy.

[0009] Dynamic Inference Network (DIN): DIN obtains the interaction relationship graph of a specific person by predicting the relationship matrix of the given person and the dynamic offset.

[0010] 3D-ResNet50-NonLocal network: It is a 3D network structure, which has good performance in extracting space-time features, and the NonLocal module is added to solve the problem of long-term space-time dependence.

[0011] RoIAlign module: the bilinear interpolation method is used to calculate the accurate value of the feature in each region of interest (RoI), and the pooling operation is used to obtain the output result, which solves the quantization error problem in RoI-Pooling.

[0012] Self-Attention Graph Pooling Network (SAG-Pool): a network that pools the graph structure through the self-attention mechanism, which has strong portability.

[0013] The specific technical scheme of the present application is as follows:

[0014] A multi-group behavior recognition method based on space-time interaction reasoning, characterized in that:

[0015] a. Use the adjacent person interaction feature extraction module to infer the spatial interaction features between individuals, deeply mine the individual appearance interaction features and spatial topological structure features, and further enrich the individual features;

[0016] b. Using the spatio-temporal clustering module, combining behavior similarity features and spatio-temporal trajectory features, to explore the relationship between individuals and divide the individuals in the whole scene into multiple small groups;

[0017] c. Using the group interaction reasoning module, combining the global trajectory graph reasoning module and the graph aggregation and pooling module, exploring the spatio-temporal interaction between different small groups and individuals, and giving different degrees of attention to different individuals, so as to obtain high-level multi-group behavior features;

[0018] The method mainly includes the following steps:

[0019] (1) Data preprocessing: frame extraction is performed on continuous video frames, and the sampled video frames are directly input into the network;

[0020] (2) Feature extraction: a 3D-ResNet50-NonLocal network is used as the backbone network to extract spatio-temporal features of the video, and then the feature information of each individual is obtained through an attention mechanism module and a RoIAlign module. The initial features of the individuals are obtained by using a fully connected layer to integrate the obtained information;

[0021] (3) Individual feature reasoning: the appearance features extracted in step (2) are used as nodes, and a GraphAttention Network and a Dynamic Inference Network are used to reason the interaction relationship between individuals;

[0022] (4) Individual behavior classification: the features reasoned in step (3) are added to the initial features extracted in step (2) to obtain individual features, which are sent to a classifier to identify the individual behavior category;

[0023] (5) Proximity person interaction feature extraction: according to the individual bounding box information, the nearest neighbors of each individual are found, and then the individual itself and the nearest neighbors are regarded as a whole to extract features. At the same time, the position and distance relationship of the individual is cascaded, and a high-dimensional feature is mapped through a fully connected layer. The individual nearest neighbor feature and the position distance feature are combined to obtain the individual interaction relationship feature;

[0024] (6) Spatio-temporal trajectory distance modeling: according to the individual trajectory information, the spatio-temporal trajectory distance between two individuals is calculated to obtain a clustering matrix based on the spatio-temporal trajectory distance;

[0025] (7) Behavior similarity modeling: according to the individual interaction relationship feature in step (5), the behavior similarity between individuals is calculated to construct a clustering matrix based on behavior similarity;

[0026] (8) Spatio-temporal clustering module: according to the clustering matrix based on the spatio-temporal trajectory distance in step (6) and the clustering matrix based on the behavior similarity in step (7), a final clustering matrix is obtained;

[0027] (9) Group clustering result: according to the clustering matrix generated in step (8), the result of group clustering is obtained;

[0028] (10) Global trajectory similarity modeling: according to the angle and length information of the local trajectory and the angle and length information of the global trajectory, the global trajectory similarity coefficient between the trajectories is obtained;

[0029] (11) Construction and reasoning of global trajectory graph: taking the individual interaction relationship feature in step (5) as a node, and the global trajectory similarity in step (10) as an edge, a global trajectory graph is constructed, and the graph is reasoned through a graph convolution network, to further mine the spatio-temporal interaction between individuals;

[0030] (12) Graph aggregation and pooling module: the initial features extracted in step (2), the individual interaction features in step (5) and the features after reasoning in step (11) are concatenated to obtain the initial features of the multi-group behavior, and then according to the group clustering result in step (9), the spatio-temporal interaction between small groups and individuals is mined using a graph aggregation network, and different degrees of attention are given to different individuals using a graph pooling network, and finally the multi-group behavior features are obtained;

[0031] (13) Multi-group behavior classification: the features mined by the graph aggregation network in step (12) are added to the final multi-group behavior features in step (12), and input into the classification layer to obtain the category of the multi-group behavior;

[0032] (14) Model training: the model constructed by (2)-(13) is trained in two steps, the first step is to extract individual features from the backbone network and directly classify behaviors, save the model parameters and input them into the second step network model; The second step is based on the first step, more information such as individual position distance information, spatio-temporal trajectory information, etc. is added, and then through graph reasoning, graph aggregation and graph pooling, the spatio-temporal interaction features between individuals and groups are further mined, and the classification results of individual behavior, group clustering and multi-group behavior are obtained.

[0033] The beneficial effects of the present application are:

[0034] (1) The 3D convolution network is used to extract the spatio-temporal features of the video, which has good effect in spatio-temporal feature mining.

[0035] (2) The neighboring person interaction feature module is used to reason the spatial interaction features between individuals, which further enriches the features of individuals in combination with the spatial topological structure.

[0036] (3) Using the spatio-temporal clustering module, combining behavior similarity features and spatio-temporal trajectory features, the association between individuals is fully mined from multiple angles, so as to more accurately divide the individuals in the entire scene into multiple small groups.

[0037] (4) Using the group interaction reasoning module, combining the global trajectory graph reasoning module and the graph aggregation and pooling module, exploring the spatio-temporal interaction between different small groups and individuals, and assigning different weights to different individuals to highlight important features, so as to obtain more accurate multi-group behavior features. BRIEF DESCRIPTION OF DRAWINGS

[0038] Figure 1 A multi-group behavior recognition framework based on spatio-temporal interaction reasoning. DETAILED DESCRIPTION

[0039] The application will be further described in detail below in combination with the drawings and examples. It is necessary to point out that the following examples are only used to further illustrate the application and cannot be understood as limiting the protection scope of the application. Those skilled in the art can make some non-essential improvements and adjustments to the specific implementation of the application according to the above description, which should still belong to the protection scope of the application.

[0040] The multi-group behavior recognition method based on spatio-temporal interaction reasoning specifically comprises the following steps:

[0041] (1) Data preprocessing

[0042] The input continuous video frames are frame-sampled, and the sampled video frames are directly input into the network for training.

[0043] (2) Feature extraction

[0044] As shown in Figure 1 , a 3D-ResNet50-NonLocal network is used as the backbone network to extract the spatio-temporal features of the video, and then the feature information of each individual is obtained through the attention mechanism module and the RoIAlign module. The initial feature f act of the individual is obtained by using a fully connected layer to integrate the obtained information.

[0045] (3) Individual feature reasoning

[0046] The appearance features extracted in step (2) are used as nodes, and the Graph Attention Network and the Dynamic Inference Network are used to reason the interaction relationship between individuals. The dimension of the features is 1024.

[0047] (4) Individual behavior classification

[0048] The features f extracted in step (3) are reasoned r-actor The initial features f extracted in step (2) act are added to obtain individual features f actor (As shown in formula (1)), which are finally sent to the classifier to identify the individual behavior category.

[0049] f actor = f act + f r-actor (1)

[0050] (5) Extraction of neighboring human interaction features

[0051] According to the bounding box information of each individual annotated by the data set, the distance between two individuals is calculated, the nearest neighbor of each individual is found, and it is required that the distance between them is less than the set threshold. Then the individual itself and the nearest neighbor are regarded as a whole to extract the feature f ai , the individual position f po and the distance relationship f di are cascaded, and mapped to high-dimensional features f p through a fully connected layer, and finally the individual nearest neighbor feature f ai and the position distance feature f p are spliced to obtain the individual interaction relationship feature f ap , and the specific process is shown in formula (2):

[0052] f ap = [f ai ; f p ] = [f ai ; f c ([f po ; f di ]) (2)

[0053] (6) Spatiotemporal trajectory distance modeling

[0054] According to the individual trajectory information, the Fréchet distance of the trajectory is calculated to reflect the spatiotemporal trajectory distance between individuals. When the spatiotemporal trajectory distance d is less than the set threshold τ i , or the proportion of the number of points n whose spatiotemporal trajectory distance is less than the set threshold to the total number of trajectory points N is greater than the set value τ r , the corresponding value of the spatiotemporal trajectory distance matrix M ST is 1, representing that the two individuals belong to the same small group.

[0055] (7) Behavior similarity modeling

[0056] According to the individual interaction relationship feature f ap, the similarity of behaviors between individuals is measured by the cosine similarity method (as shown in equation (3)), and when the value of behavior similarity is greater than the set threshold, the corresponding value of the behavior similarity matrix M ap is 1, representing that the two individuals belong to the same small group.

[0057]

[0058] where, and are the features of the ith individual and the jth individual. and are learnable linear transformations, and are weight matrices, and are weight vectors.

[0059] (8) Spatio-temporal clustering module

[0060] Multiply the clustering matrix based on spatio-temporal trajectory distance of step (6) and the clustering matrix based on behavior similarity of step (7) to obtain the final clustering matrix, as shown in equation (4):

[0061] M=M ap ⊙M ST (4)

[0062] (9) Spatio-temporal clustering result

[0063] According to the clustering matrix generated in step (8), the group clustering result is obtained.

[0064] (10) Global trajectory similarity modeling

[0065] According to the individual bounding box information, the trajectory of each individual is obtained (here taking trajectories P and Q as examples), and the local trajectory angle θ i and length feature l i , and global trajectory angle θ global and length feature l global are calculated from the trajectory of the individual, and the global trajectory similarity coefficient Sim(P, Q) is obtained by comprehensive consideration, as shown in the following equation:

[0066]

[0067]

[0068]

[0069] Sim(P,Q)=Softmax(-lnDis(P,Q)) (8)

[0070] (11) Construction and reasoning of global trajectory graph

[0071] The individual interaction feature after step (5) is taken as a node, and the global trajectory similarity of step (10) is taken as an edge to construct a global trajectory graph, and the graph is reasoned through a graph convolution network to further mine the spatio-temporal interaction between individuals, and the feature after reasoning is denoted as f apt .

[0072] (12) Graph aggregation and pooling module

[0073] The initial feature f act extracted in step (2), the individual interaction feature f ap of step (5), and the feature f apt after reasoning of step (11) are concatenated to obtain the initial feature f g of multi-group behavior, and then according to the group clustering result of step (9), a graph aggregation network is used to mine the spatio-temporal interaction between small groups and individuals, and the output f agg of the graph aggregation network is added to the initial feature f g of multi-group behavior to obtain f res , which is input into a graph pooling network to calculate a self-attention score S, and different individuals are given different degrees of attention, and finally the multi-group behavior feature f r-group is obtained, which is specifically shown in the following formula:

[0074] f r-group =m(f res οS)=m((f agg +f g )οS) (9)

[0075] Wherein, m(·) is a max-pooling operation, and o represents a dot product operation.

[0076] (13) Multi-group behavior classification

[0077] The feature f res mined by the graph aggregation network of step (12) is added to the final multi-group behavior feature f r-group of step (12) to obtain f group , which is input into a classification layer to obtain the category R of multi-group behavior.

[0078] R=f g-classifier (f group )=f g-classifier (f res +f r-group ) (10)

[0079] (14) Model training

[0080] The model training constructed by (2)-(13) is divided into two steps. The first step is to extract individual features by the backbone network and directly classify behaviors. The model parameters are saved and input into the second step network model. The second step adds more information such as individual position distance information, space-time trajectory information, etc. on the basis of the first step. Through graph reasoning, graph aggregation and graph pooling, the space-time interaction features between individuals and groups are further mined, and the classification results of individual behavior, group clustering and multi-group behavior are obtained.

Claims

1. A multi-group behavior recognition method based on spatio-temporal interaction reasoning, characterized in that: a.using a neighboring person interaction feature extraction module to reason the spatial interaction features between individuals, deeply mining individual appearance interaction features and spatial topological structure features, and further enriching individual features; b.using a spatio-temporal clustering module, combining behavior similarity features and spatio-temporal trajectory features to explore the relationship between individuals and divide the individuals in the whole scene into multiple small groups; c.using a group interaction reasoning module, combining a global trajectory graph reasoning module and a graph aggregation and pooling module to explore the spatio-temporal interaction between different small groups and individuals, and giving different degrees of attention to different individuals, thereby obtaining high-level multi-group behavior features; the method mainly includes the following steps: (1) data preprocessing: frame extraction is performed on continuous video frames, and the sampled video frames are directly input into the network; (2) feature extraction: a 3D-ResNet50-NonLocal network is used as the backbone network to extract spatio-temporal features of the video, and then the attention mechanism module and the RoIAlign module are used to obtain the feature information of each individual, and the initial features of the individuals are obtained by using the full connection layer to integrate the obtained information; (3) individual feature reasoning: the appearance features extracted in step (2) are used as nodes, and the GraphAttentionNetwork and the DynamicInferenceNetwork are used to reason the interaction relationship between individuals; (4) individual behavior classification: the features reasoned in step (3) and the initial features extracted in step (2) are added to obtain individual features, which are sent to a classifier to recognize individual behavior categories; (5) neighboring person interaction feature extraction: according to the individual bounding box information, the nearest neighbors of each individual are found, then the individual itself and the nearest neighbors are regarded as a whole to extract features, and at the same time the individual position and distance relationship are cascaded, which are mapped to high-dimensional features through a full connection layer, and the individual interaction relationship features are obtained by combining the individual nearest neighbor features and the position distance features; (6) spatio-temporal trajectory distance modeling: according to the individual trajectory information, the spatio-temporal trajectory distance between two individuals is calculated to obtain a clustering matrix based on the spatio-temporal trajectory distance; (7) behavior similarity modeling: the behavior similarity between individuals is calculated according to the individual interaction relationship features in step (5) to construct a clustering matrix based on behavior similarity; (8) spatio-temporal clustering module: according to the clustering matrix based on the spatio-temporal trajectory distance in step (6) and the clustering matrix based on the behavior similarity in step (7), a final clustering matrix is obtained; (9) group clustering result: the group clustering result is obtained according to the clustering matrix generated in step (8); (10) global trajectory similarity modeling: according to the angle and length information of the local trajectory and the angle and length information of the global trajectory, the global trajectory similarity coefficient between trajectories is obtained. ​ ​ ​ ​ ​ ​ ​ ​ ​ ​ ​ ​ ​ ​ (11) Construction and reasoning of global trajectory graph: taking the individual interaction features in step (5) as nodes and the global trajectory similarity in step (10) as edges, a global trajectory graph is constructed, and the graph is reasoned through a graph convolution network to further mine the spatio-temporal interaction between individuals; (12) Graph aggregation and pooling module: cascade the initial features extracted in step (2), the individual interaction features in step (5) and the features reasoned in step (11) to obtain the initial features of multi-group behavior, then use the graph aggregation network to mine the spatio-temporal interaction between small groups and individuals according to the group clustering results in step (9), and use the graph pooling network to give different degrees of attention to different individuals, and finally obtain the multi-group behavior features; (13) Multi-group behavior classification: add the features mined by the graph aggregation network in step (12) to the final multi-group behavior features in step (12), and input them into the classification layer to obtain the category of multi-group behavior; (14) Model training: the model constructed by (2)-(13) is trained in two steps. The first step is to extract individual features from the backbone network and directly classify behaviors. The model parameters are saved and input into the second step network model. The second step adds individual position distance information and spatio-temporal trajectory information on the basis of the first step. Through graph reasoning, graph aggregation and graph pooling, the spatio-temporal interaction features between individuals and groups are further mined to obtain the classification results of individual behavior, group clustering and multi-group behavior.

2. The spatio-temporal interaction reasoning based multi-group behavior recognition method of claim 1, wherein In step (2), the channel attention mechanism Squeeze-and-excitation Networks is added to the backbone network 3D-ResNet50-NonLocal to give different degrees of attention to different channels. On the basis of extracting spatio-temporal features, the channels that contribute more to behavior recognition are given greater weight allocation, and the channels that have little effect are inhibited.

3. The spatio-temporal interaction reasoning based multi-group behavior recognition method of claim 1, wherein In step (5), in order to extract more rich individual interaction features, the features of the nearest neighbor and the position distance features are used as supplements to the individual features. Specifically, the distance between individuals is calculated through the individual bounding box information, the nearest neighbor of each individual is found, and it is required that the distance between them is less than the set threshold. Then the individual itself and the nearest neighbor are regarded as a whole to extract features, and the distance between individuals and the coordinates of individual bounding boxes are concatenated. Finally, the individual nearest neighbor features and the position distance features are combined to obtain the individual interaction features.

4. The spatio-temporal interaction reasoning based multi-group behavior recognition method of claim 1, wherein In step (6), the spatio-temporal trajectory measurement method is used as a condition for group clustering. Specifically, the trajectory information of individuals is obtained according to the individual bounding box, and the Fréchet distance of the trajectory is calculated to reflect the spatio-temporal trajectory distance between individuals. When the spatio-temporal trajectory distance is less than the set threshold, or the proportion of the number of points with spatio-temporal trajectory distance less than the set threshold to the total number of trajectory points is greater than the set value, the corresponding value of the spatio-temporal trajectory distance matrix is 1, representing that the two individuals belong to the same small group.

5. The spatio-temporal interaction reasoning based multi-group behavior recognition method of claim 1, wherein In step (7), the method of behavior similarity is used as a condition for group clustering. Specifically, in the construction of a behavior similarity-based clustering matrix, the cosine similarity method is used to calculate the similarity between behaviors. When the behavior similarity value is greater than the set threshold, the corresponding value of the behavior similarity matrix is 1, representing that the two individuals belong to the same small group.

6. The spatio-temporal interaction reasoning based multi-group behavior recognition method of claim 1, wherein In step (8), the final group clustering result needs to consider both the space-time trajectory features and the behavior similarity features, i.e., multiplying the clustering matrices of steps (6) and (7) as the final clustering determination condition.

7. The spatio-temporal interaction reasoning based multi-group behavior recognition method of claim 1, wherein In step (10), the individual's trajectory is obtained according to the individual's bounding box information. The local trajectory angle and length features, as well as the global trajectory angle and length features, are calculated from the individual's trajectory. The global trajectory similarity coefficient is obtained to reflect the relationship between individuals.

8. The spatio-temporal interaction reasoning based multi-group behavior recognition method of claim 1, wherein In step (11), in the process of constructing the global trajectory graph, the individual interaction features of step (5) are used as nodes, and the global trajectory similarity coefficient is used as the edge information of the graph. The graph convolution network is used for interaction reasoning.

9. The spatio-temporal interaction reasoning based multi-group behavior recognition method of claim 1, wherein In step (12), according to the group clustering result of step (9), the graph aggregation network is used to mine the space-time interaction information between small groups and individuals. The self-attention graph pooling network is used to generate attention scores through graph convolution. According to the attention scores, the nodes are sorted, and the top k nodes are selected to input into the maximum pooling network to obtain the final high-level multi-group behavior features.

Images