Multi-group behavior recognition method based on space-time interaction reasoning

Abstract

The invention provides a multi-group behavior recognition method based on space-time interaction reasoning, and mainly relates to the problem of how to effectively mine space-time interaction so as to carry out individual behavior, group clustering and multi-group behavior recognition. According to the method, a space-time interaction relationship reasoning strategy is put forward to pay attention to interaction relationships between individuals and between groups, meanwhile, features such as appearances, position distances and space-time trajectories are considered, and the accuracy of behavior recognition is improved. Firstly, the interaction relation between individuals is explored by analyzing features of nearest neighbors of the individuals and position distance features. Secondly, dividing the individuals into a plurality of small groups in combination with behavior similarity and spatio-temporal trajectory characteristics; and finally, a group interaction reasoning module is constructed, interaction among different small groups and individuals is mined, and the individuals having small influence on group behaviors are excluded according to importance. According to the method, the space-time interaction characteristics of the individuals and the groups are fully considered, and the problem that space-time interaction characteristics are not fully mined in behavior recognition is solved.

Description

Technical Field

[0001] The present invention relates to individual behavior, group clustering and multi-group behavior recognition problems in the field of deep learning, and in particular to a multi-group behavior recognition method based on spatiotemporal interactive reasoning. Background Art

[0002] Video-based human activity understanding is a core research area of ​​computer vision, and has broad application prospects in areas such as surveillance systems and social scene understanding. Multi-group behavior recognition is a relatively complex area in the study of human activity understanding. Most existing studies extract individual features through convolutional neural networks, and then obtain group-level features through further reasoning and pooling operations. Unlike individual behavior recognition, multi-group behavior recognition focuses more on the mining of interactive relationships. With the development of graph neural networks, related studies have adopted graph convolutional networks to infer the potential interactive relationships between individuals, in order to enrich group features and improve the accuracy of behavior recognition.

[0003] However, the existing behavior recognition methods are not deep enough in the mining of spatiotemporal interaction relationships, resulting in inaccurate group-level features. Therefore, this patent focuses on the mining of spatiotemporal interaction features, while considering features such as appearance, position distance, spatiotemporal trajectory, and using a graph pooling network to pay different degrees of attention to different individuals, thereby obtaining high-level group behavior features and improving the accuracy of behavior recognition. First, a 3D convolutional network is used to extract the spatiotemporal features of the video, and then the attention mechanism and RoIAlign module are used to extract the features of each individual. Secondly, the Graph Attention Network and Dynamic Inference Network are used to infer the interaction relationship between individuals and obtain the category of individual behavior. Then, the interaction relationship between individuals is explored by analyzing the features of the individual's nearest neighbors and the position distance features. Then, the individuals are divided into multiple small groups by combining the behavioral similarity features and spatiotemporal trajectory features. Finally, by constructing a group interaction reasoning module, the interactions between different small groups and individuals are mined, and people who have little influence on group behavior are excluded according to their importance. After entering the classifier, the multi-group behavior recognition results are obtained. Summary of the invention

[0004] The purpose of the present invention is to provide a multi-group behavior recognition method based on spatiotemporal interaction reasoning, which fully considers the spatiotemporal interaction characteristics of individuals and groups, uses graph neural networks to deeply mine spatiotemporal interaction characteristics, and considers features such as appearance, position distance, and spatiotemporal trajectory. It also uses a self-attention graph pooling network to give different degrees of attention to different individuals, thereby enriching the group-level features and solving the problem of insufficient mining of spatiotemporal interaction features in behavior recognition.

[0005] For the convenience of explanation, first introduce the following concepts:

[0006] Graph: A graph structure consisting of nodes and edges connecting nodes.

[0007] Graph Convolutional Network (GCN): The purpose of GCN is to extract the spatial features of topological graphs, and the data processed is a graph structure, that is, a non-Euclidean structure.

[0008] Graph Attention Network (GAT): GAT can flexibly learn the attention weights between nodes through parameterized operations based on self-attention strategies.

[0009] Dynamic Inference Network (DIN): DIN obtains the interaction relationship graph of a specific person by predicting the relationship matrix of a given person and the dynamic offset.

[0010] 3D-ResNet50-NonLocal network: It is a 3D network structure that performs well in extracting spatiotemporal features. The NonLocal module is added to solve the problem of long-term spatiotemporal dependency.

[0011] RoIAlign module: uses bilinear interpolation to calculate the exact value of the features in each region of interest (RoI), and uses pooling operation to obtain the output result, solving the problem of quantization error in RoI-Pooling.

[0012] Self-Attention Graph Pooling Network (SAG-Pool): A network that pools graph structures through a self-attention mechanism, which has strong portability.

[0013] The present invention specifically adopts the following technical solutions:

[0014] A multi-group behavior recognition method based on spatiotemporal interactive reasoning, characterized by:

[0015] a. Use the neighboring person interaction feature extraction module to infer the spatial interaction features between individuals, deeply explore the individual appearance interaction features and spatial topological structure features, and further enrich the individual features;

[0016] b. Use the spatiotemporal clustering module to combine behavioral similarity features and spatiotemporal trajectory features to explore the relationship between individuals and divide individuals in the entire scene into multiple small groups;

[0017] c. Use the group interaction reasoning module, combined with the global trajectory graph reasoning module and the graph aggregation and pooling module, to explore the spatiotemporal interactions between different small groups and individuals, and pay different degrees of attention to different individuals, so as to obtain high-level multi-group behavior characteristics;

[0018] The method mainly includes the following steps:

[0019] (1) Data preprocessing: Continuous video frames are sampled and the sampled video frames are directly input into the network;

[0020] (2) Feature extraction: The 3D-ResNet50-NonLocal network is used as the backbone network to extract the spatiotemporal features of the video. The feature information of each individual is obtained through the attention mechanism module and the RoIAlign module. The fully connected layer is used to integrate the obtained information to obtain the initial features of the individual.

[0021] (3) Individual feature reasoning: The appearance features extracted in step (2) are used as nodes, and the GraphAttention Network and Dynamic Inference Network are used to infer the interaction relationship between individuals;

[0022] (4) Individual behavior classification: The features inferred in step (3) are added to the initial features extracted in step (2) to obtain individual features, which are then fed into the classifier to identify the individual behavior category.

[0023] (5) Neighbor interaction feature extraction: Based on the individual annotation box information, find the nearest neighbor of each individual, then extract features from the individual and its nearest neighbor as a whole. At the same time, concatenate the individual position and distance relationship, map them into high-dimensional features through a fully connected layer, and combine the individual nearest neighbor features and position distance features to obtain the individual interaction relationship features.

[0024] (6) Spatiotemporal trajectory distance modeling: Based on the individual trajectory information, the spatiotemporal trajectory distance between two individuals is calculated to obtain a clustering matrix based on the spatiotemporal trajectory distance;

[0025] (7) Behavior similarity modeling: Calculate the behavioral similarity between individuals based on the individual interaction relationship characteristics in step (5) and construct a clustering matrix based on behavioral similarity;

[0026] (8) Spatiotemporal clustering module: According to the clustering matrix based on spatiotemporal trajectory distance and the clustering matrix based on behavior similarity in step (6) and step (7), the final clustering matrix is ​​obtained;

[0027] (9) Group clustering results: According to the clustering matrix generated in step (8), the group clustering results are obtained;

[0028] (10) Global trajectory similarity modeling: Based on the angle and length information of the local trajectory and the angle and length information of the global trajectory, the global trajectory similarity coefficient between the trajectories is obtained;

[0029] (11) Construction and reasoning of the global trajectory graph: The individual interaction relationship features after step (5) are used as nodes, and the global trajectory similarity of step (10) is used as edges to construct a global trajectory graph. The graph is reasoned through a graph convolutional network to further explore the spatiotemporal interactions between individuals.

[0030] (12) Graph aggregation and pooling module: The initial features extracted in step (2), the individual interaction features in step (5), and the features inferred in step (11) are concatenated to obtain the initial features of multi-group behavior. Then, based on the group clustering results in step (9), a graph aggregation network is used to mine the spatiotemporal interactions between small groups and individuals, and a graph pooling network is used to give different degrees of attention to different individuals, ultimately obtaining multi-group behavior features.

[0031] (13) Multi-group behavior classification: Add the features mined by the graph aggregation network in step (12) to the final multi-group behavior features in step (12), input them into the classification layer, and obtain the categories of multi-group behaviors;

[0032] (14) Model training: The training of the model constructed by (2)-(13) is divided into two steps. In the first step, the backbone network extracts individual features and directly classifies the behavior. After saving the model parameters, the model is input into the network model of the second step. In the second step, based on the first step, more information is added, such as individual location distance information, spatiotemporal trajectory information, etc., and the spatiotemporal interaction characteristics between individuals and groups are further explored through graph reasoning, graph aggregation, and graph pooling, so as to obtain the classification results of individual behavior, group clustering, and multi-group behavior.

[0033] The beneficial effects of the present invention are:

[0034] (1) Using 3D convolutional networks to extract the spatiotemporal features of videos has a good effect in mining spatiotemporal features.

[0035] (2) The neighbor interaction feature module is used to infer the spatial interaction features between individuals, and combined with the spatial topological structure, the individual characteristics are further enriched.

[0036] (3) Using the spatiotemporal clustering module, combined with behavioral similarity features and spatiotemporal trajectory features, the associations between individuals are fully explored from multiple perspectives, thereby more accurately dividing the individuals in the entire scene into multiple small groups.

[0037] (4) The group interaction reasoning module is used in combination with the global trajectory graph reasoning module and the graph aggregation and pooling module to explore the spatiotemporal interactions between different small groups and between individuals, and different weights are assigned to different individuals to highlight important features, thereby obtaining more accurate multi-group behavior characteristics. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] Figure 1 Diagram of the framework for multi-group behavior recognition based on spatiotemporal interaction reasoning. DETAILED DESCRIPTION

[0039] The present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It is necessary to point out that the following embodiments are only used to further illustrate the present invention and cannot be understood as limiting the scope of protection of the present invention. Technical personnel familiar with the relevant field of technology may make some non-essential improvements and adjustments to the present invention based on the above invention content for specific implementation, which should still fall within the scope of protection of the present invention.

[0040] The multi-group behavior recognition method based on spatiotemporal interactive reasoning specifically includes the following steps:

[0041] (1) Data preprocessing

[0042] The input continuous video frames are sampled and the sampled video frames are directly input into the network for training.

[0043] (2) Feature extraction

[0044] like Figure 1 As shown in the figure, the 3D-ResNet50-NonLocal network is used as the backbone network to extract the spatiotemporal features of the video, and then the feature information of each individual is obtained through the attention mechanism module and the RoIAlign module. The fully connected layer is used to integrate the obtained information to obtain the initial feature f of the individual. act .

[0045] (3) Individual characteristic reasoning

[0046] The appearance features extracted in step (2) are used as nodes, and the Graph Attention Network and Dynamic Inference Network are used to infer the interaction relationship between individuals. The dimension of the feature is 1024.

[0047] (4) Classification of individual behaviors

[0048] The feature f inferred by step (3) r-actor The initial feature f extracted in step (2) act Add together to get the individual feature f actor (As shown in formula (1)), and finally sent to the classifier to identify the individual behavior category.

[0049] f actor =f act +f r-actor (1)

[0050] (5) Neighboring People Interaction Feature Extraction

[0051] According to the bounding box information of each individual in the data set, the distance between two individuals is calculated, and the nearest neighbor of each individual is found, and the distance between them is required to be less than the set threshold. Then the individual itself and the nearest neighbor are regarded as a whole to extract the feature f ai , and at the same time the individual position f po And the distance relationship f di Cascade and map to high-dimensional features f through the fully connected layer p , and finally concatenate the individual nearest neighbor features f ai And the position distance feature f p , and obtain the individual interaction relationship characteristics f ap , the specific process is shown in formula (2):

[0052] f ap =[f ai ;f p ]=[f ai ;f c ([f po ;f di ])] (2)

[0053] (6) Spatial-temporal trajectory distance modeling

[0054] According to the individual trajectory information, the Fréchet distance of the trajectory is calculated to reflect the temporal and spatial trajectory distance between individuals. When the temporal and spatial trajectory distance d is less than the set threshold τ i When the space-time trajectory distance is less than the set threshold, the ratio of the number of points n to the total number of trajectory points N is greater than the set value τ r When, the space-time trajectory distance matrix M ST The corresponding value of is 1, indicating that the two individuals belong to the same small group.

[0055] (7) Behavioral Similarity Modeling

[0056] According to the individual interaction relationship feature f obtained in step (5) ap, the cosine similarity method is used to measure the similarity of behaviors between individuals (as shown in formula (3)). When the value of the behavior similarity is greater than the set threshold, the behavior similarity matrix M ap The corresponding value of is 1, indicating that the two individuals belong to the same small group.

[0057]

[0058] in, and are the characteristics of the i-th and j-th individuals. and is a learnable linear transformation, and is the weight matrix, and is the weight vector.

[0059] (8) Spatiotemporal clustering module

[0060] The clustering matrix based on spatiotemporal trajectory distance and the clustering matrix based on behavioral similarity in steps (6) and (7) are multiplied to obtain the final clustering matrix, as shown in formula (4):

[0061] M=M ap ⊙M ST (4)

[0062] (9) Spatiotemporal clustering results

[0063] According to the clustering matrix generated in step (8), the group clustering result is obtained.

[0064] (10) Global trajectory similarity modeling

[0065] According to the individual annotation box information, the trajectory of each individual is obtained (here we take trajectory P, Q as an example), and the local trajectory angle θ is calculated from the individual trajectory i and length feature l i , and the global trajectory angle θ global and length feature l global , the global trajectory similarity coefficient Sim(P,Q) is obtained comprehensively, as shown in the following formula:

[0066]

[0067]

[0068]

[0069] Sim(P,Q)=Softmax(-lnDis(P,Q)) (8)

[0070] (11) Construction and reasoning of global trajectory graph

[0071] The individual interaction relationship features after step (5) are used as nodes, and the global trajectory similarity of step (10) is used as edges to construct a global trajectory graph. The graph is then inferred through a graph convolutional network to further explore the spatiotemporal interactions between individuals. The inferred features are recorded as f apt .

[0072] (12) Graph aggregation and pooling module

[0073] The initial feature f extracted in step (2) act , the individual interaction feature f in step (5) ap And the feature f after inference in step (11) apt Cascade to obtain the initial features f of multiple group behaviors g , and then use the graph aggregation network to mine the spatiotemporal interactions between small groups and individuals based on the group clustering results of step (9), and then use the output of the graph aggregation network f agg Plus the initial features of multiple group behaviors f g Get f res , input the graph pooling network, calculate the self-attention score S, give different degrees of attention to different individuals, and finally obtain the multi-group behavior characteristics f r-group , as shown in the following formula:

[0074] f r-group =m(f res οS)=m((f agg +f g )οS) (9)

[0075] Among them, m(·) is the maximum pooling operation, and ο represents the dot product operation.

[0076] (13) Multi-group behavior classification

[0077] The feature f mined by the graph aggregation network in step (12) res The final multi-group behavior feature f in step (12) r-group Add them together to get f group , input the classification layer, and obtain the category R of multiple group behaviors.

[0078] R = f g-classifier (f group ) = f g-classifier (f res +f r-group ) (10)

[0079] (14) Model training

[0080] The training of the model constructed by (2)-(13) is divided into two steps. In the first step, the backbone network extracts individual features and directly performs behavior classification. After saving the model parameters, it is input into the network model of the second step. In the second step, based on the first step, more information is added, such as individual location distance information, spatiotemporal trajectory information, etc., and then through graph reasoning, graph aggregation and graph pooling, the spatiotemporal interaction characteristics between individuals and groups are further explored to obtain the classification results of individual behavior, group clustering and multi-group behavior.

Claims

1. A multi-group behavior recognition method based on spatiotemporal interactive reasoning, characterized by: a. Use the neighboring person interaction feature extraction module to infer the spatial interaction features between individuals, deeply explore the individual appearance interaction features and spatial topological structure features, and further enrich the individual features; b. Use the spatiotemporal clustering module to combine behavioral similarity features and spatiotemporal trajectory features to explore the relationship between individuals and divide individuals in the entire scene into multiple small groups; c. Use the group interaction reasoning module, combined with the global trajectory graph reasoning module and the graph aggregation and pooling module, to explore the spatiotemporal interactions between different small groups and individuals, and pay different degrees of attention to different individuals, so as to obtain high-level multi-group behavior characteristics; The method mainly includes the following steps: (1) Data preprocessing: Continuous video frames are sampled and the sampled video frames are directly input into the network; (2) Feature extraction: The 3D-ResNet50-NonLocal network is used as the backbone network to extract the spatiotemporal features of the video. The feature information of each individual is obtained through the attention mechanism module and the RoIAlign module. The fully connected layer is used to integrate the obtained information to obtain the initial features of the individual. (3) Individual feature reasoning: The appearance features extracted in step (2) are used as nodes, and the interaction relationship between individuals is inferred using GraphAttentionNetwork and DynamicInferenceNetwork; (4) Individual behavior classification: The features inferred in step (3) are added to the initial features extracted in step (2) to obtain individual features, which are then fed into the classifier to identify the individual behavior category. (5) Neighbor interaction feature extraction: Based on the individual annotation box information, find the nearest neighbor of each individual, then extract features from the individual and its nearest neighbor as a whole. At the same time, concatenate the individual position and distance relationship, map them into high-dimensional features through a fully connected layer, and combine the individual nearest neighbor features and position distance features to obtain the individual interaction relationship features. (6) Spatiotemporal trajectory distance modeling: Based on the individual trajectory information, the spatiotemporal trajectory distance between two individuals is calculated to obtain a clustering matrix based on the spatiotemporal trajectory distance; (7) Behavior similarity modeling: Calculate the behavioral similarity between individuals based on the individual interaction relationship characteristics in step (5) and construct a clustering matrix based on behavioral similarity; (8) Spatiotemporal clustering module: According to the clustering matrix based on spatiotemporal trajectory distance and the clustering matrix based on behavior similarity in step (6) and step (7), the final clustering matrix is ​​obtained; (9) Group clustering results: According to the clustering matrix generated in step (8), the group clustering results are obtained; (10) Global trajectory similarity modeling: Based on the angle and length information of the local trajectory and the angle and length information of the global trajectory, the global trajectory similarity coefficient between the trajectories is obtained; (11) Construction and reasoning of the global trajectory graph: The individual interaction relationship features after step (5) are used as nodes, and the global trajectory similarity of step (10) is used as edges to construct a global trajectory graph. The graph is reasoned through a graph convolutional network to further explore the spatiotemporal interactions between individuals. (12) Graph aggregation and pooling module: The initial features extracted in step (2), the individual interaction features in step (5), and the features inferred in step (11) are concatenated to obtain the initial features of multi-group behavior. Then, based on the group clustering results in step (9), a graph aggregation network is used to mine the spatiotemporal interactions between small groups and individuals, and a graph pooling network is used to give different degrees of attention to different individuals, ultimately obtaining multi-group behavior features. (13) Multi-group behavior classification: Add the features mined by the graph aggregation network in step (12) to the final multi-group behavior features in step (12), input them into the classification layer, and obtain the categories of multi-group behaviors; (14) Model training: The training of the model constructed by (2)-(13) is divided into two steps. In the first step, the backbone network extracts individual features and directly classifies the behavior. After saving the model parameters, the model is input into the network model of the second step. In the second step, based on the first step, more information is added, such as individual location distance information, spatiotemporal trajectory information, etc., and the spatiotemporal interaction characteristics between individuals and groups are further explored through graph reasoning, graph aggregation, and graph pooling, so as to obtain the classification results of individual behavior, group clustering, and multi-group behavior.

2. The multi-group behavior recognition method based on spatiotemporal interactive reasoning as claimed in claim 1, characterized in that In step (2), a channel attention mechanism Squeeze-and-excitation Networks is added to the backbone network 3D-ResNet50-NonLocal to pay different degrees of attention to different channels. On the basis of extracting spatiotemporal features, it better allocates greater weight to channels that contribute more to behavior recognition and suppresses channels that have little effect.

3. The multi-group behavior recognition method based on spatiotemporal interactive reasoning as claimed in claim 1, characterized in that In step (5), in order to extract richer individual interaction relationship features, the features of the nearest neighbors and the position distance features are used as supplements to the individual features. Specifically, the distance between individuals is calculated through the individual annotation box information, and the nearest neighbors of each individual are found, and the distance between them is required to be less than the set threshold. Then, the individual itself and the nearest neighbors are regarded as a whole to extract features, and the distance between individuals and the coordinates of the individual annotation boxes are cascaded. Then, they are mapped into high-dimensional features through a fully connected layer, and finally, the individual nearest neighbor features and the position distance features are combined to obtain the individual interaction relationship features.

4. The multi-group behavior recognition method based on spatiotemporal interactive reasoning as claimed in claim 1, characterized in that In step (6), the spatiotemporal trajectory measurement method is used as a condition for group clustering. Specifically, the individual trajectory information is obtained based on the individual annotation box, and the spatiotemporal trajectory distance between individuals is reflected by calculating the Fréchet distance of the trajectory. When the spatiotemporal trajectory distance is less than the set threshold, or the proportion of points with spatiotemporal trajectory distance less than the set threshold to the total number of trajectory points is greater than the set value, the corresponding value of the spatiotemporal trajectory distance matrix is ​​1, indicating that the two individuals belong to the same small group.

5. The multi-group behavior recognition method based on spatiotemporal interactive reasoning as claimed in claim 1, characterized in that In step (7), the behavioral similarity method is used as a condition for group clustering. Specifically, in constructing a clustering matrix based on behavioral similarity, the cosine similarity method is used to calculate the behavioral similarity between individuals. When the value of the behavioral similarity is greater than the set threshold, the corresponding value of the behavioral similarity matrix is ​​1, indicating that the two individuals belong to the same small group.

6. The multi-group behavior recognition method based on spatiotemporal interactive reasoning as claimed in claim 1, characterized in that In step (8), the final group clustering result needs to consider both the spatiotemporal trajectory characteristics and the behavioral similarity characteristics, that is, multiplying the clustering matrices of step (6) and step (7) as the final clustering judgment condition.

7. The multi-group behavior recognition method based on spatiotemporal interactive reasoning as claimed in claim 1, characterized in that In step (10), the trajectory of each individual is obtained based on the individual annotation box information, and the local trajectory angle and length features, as well as the global trajectory angle and length features, are calculated from the individual trajectory. The global trajectory similarity coefficient is obtained by combining them to reflect the connection between individuals.

8. The multi-group behavior recognition method based on spatiotemporal interactive reasoning as claimed in claim 1, characterized in that In step (11), during the construction of the global trajectory graph, the individual interaction features of step (5) are used as nodes, and the global trajectory similarity coefficient is used as the edge information of the graph, which is sent to the graph convolutional network for interactive reasoning.

9. The multi-group behavior recognition method based on spatiotemporal interactive reasoning as claimed in claim 1, characterized in that In step (12), based on the group clustering results of step (9), a graph aggregation network is used to mine the spatiotemporal interaction information between small groups and individuals. At the same time, a self-attention graph pooling network is used to generate attention scores through graph convolution. The nodes are sorted according to the attention scores, and the top k nodes are selected and sent to the maximum pooling network to obtain the final high-level multi-group behavior features.

Images