A group behavior recognition method based on dynamic and static features and multiple interaction networks

Abstract

The application discloses a kind of group behavior identification method based on dynamic and static features and multiple interaction networks, it is related to computer vision technical field, including to original video data and skeleton node data are preprocessed, obtain the dynamic and static features of participant;With extraction coarse-grained key figure information, and restore fine-grained figure information to obtain multi-level figure features;Application Sptial-Transformer, Temporal-Transformer obtains complex space-time interaction features;Through data fusion, the identification result is predicted;Utilize dynamic feature to provide supplement to static feature, comprehensively, carefully analyze the group behavior of participant;In order to emphasize key figure information, key figure information is used as consistency constraint, guide the focus of space-time interaction network, enhance the identification ability of model to key figure space-time dynamic.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computer vision, and particularly relates to a group behavior recognition method based on dynamic and static features and a multiple interaction network. BACKGROUND

[0002] Group behavior recognition is a promising research direction in the field of computer vision due to its versatility in application. The purpose of group behavior recognition is to identify the behavior of a group of people in a specific scene, which is widely used in intelligent monitoring, content-based video retrieval, automatic video analysis and annotation, and motion analysis, and has a positive driving effect on many industries such as transportation and criminal investigation.

[0003] Group behavior recognition methods can be divided into traditional methods and deep learning methods. Traditional group behavior recognition methods have many limitations and generally have average results. In recent years, due to the development of deep learning technology, significant progress has been made in this field. Group behavior recognition methods based on deep learning can be divided into three categories: methods based on hierarchical spatio-temporal networks, methods based on deep learning relationship models, and methods based on attention mechanisms.

[0004] 1) Methods based on hierarchical spatio-temporal networks

[0005] Methods based on hierarchical spatio-temporal networks first use convolutional neural networks to extract local person features, and then use hierarchical recurrent neural networks to gradually fuse feature sequences at different levels, finally obtaining global features of the entire group for group behavior recognition. This type of method only extracts different levels of features in stages, and uses high-level features to determine group behavior, ignoring the interaction between participants and between participants and the group.

[0006] 2) Methods based on deep learning relationship models

[0007] Methods based on deep learning relationship models mostly use deep learning technology to model the interaction between participants and between participants and the group in a multi-person scene. This type of method uses deep learning relationship networks to extract participant interaction information, making participant features more complete. However, different types of interaction features need to be extracted using targeted deep learning relationship models to ensure that more detailed and comprehensive participant interaction features are captured.

[0008] 3) Methods based on attention mechanisms

[0009] The method based on the attention mechanism is proposed from 2020 by Google Vision Transformer (ViT), the model can not need convolution, directly utilize the Transformer to the image block sequence carries out classification, and obtains the result comparable with the current optimal convolution network, but compared with the convolution neural network, the calculation resources required for training are greatly reduced, and the challenge of high complexity and large calculation workload of group behavior recognition task is solved to a certain extent. In the existing scheme that applies the attention mechanism to process the unequal interaction relationship in the group activity, most of them cannot adjust the attention range according to the downstream task, and lack of flexibility and adaptability.

[0010] In view of the problems in the prior art that the static features of participants are insufficient to fully represent the behaviors of the participants, the existing method of extracting dynamic features of participants using optical flow has high complexity and is unstable, and the model simply uses the attention mechanism to highlight the key figures, which cannot flexibly adjust the number of key figures, the present application aims to provide a group behavior recognition method based on dynamic and static features and multiple interaction networks. SUMMARY

[0011] In order to make up for the deficiencies of the prior art, the present application provides a group behavior recognition method based on dynamic and static features and multiple interaction networks, which uses dynamic features to supplement static features, comprehensively and meticulously analyzes the group behavior of participants, extracts coarse-grained key figure information and restores fine-grained figure information, and excludes the interference of non-key figure interaction on the correct action recognition of the model, in order to emphasize key figure information, the key figure information is used as a consistency constraint to guide the attention point of the space-time interaction network, and the recognition ability of the model for the space-time dynamics of key figures is enhanced.

[0012] To achieve the above object, the present application provides the following technical scheme:

[0013] A group behavior recognition method based on dynamic and static features and multiple interaction networks, comprising the following steps:

[0014] (1) preprocessing the original video data and skeleton node data;

[0015] (2) extracting multi-scale features from the preprocessed video data using Inception-v3, and then extracting static features of all participants in each frame through RoIAlign;

[0016] (3) inputting the preprocessed skeleton node data into a spatial graph convolution network and a temporal graph convolution network in sequence to obtain the space-time motion features of the internal joint nodes of the human body, and inputting the space-time motion features of the internal joint nodes of the human body into a TemporalAttention module to extract the dynamic features of the participants as a whole;

[0017] (4) After changing the dimension of the dynamic features of the whole participant so that their dimension is the same as the static feature dimension, the static feature is fused with the dynamic features of the whole participant after the dimension change to obtain the dynamic and static features;

[0018] (5) Use multiple coarse-grained key person extraction modules to extract information from multiple key persons, and then use multiple fine-grained person information restoration modules to restore information for all key persons. After the multiple fine-grained person information restoration modules, add a GCN layer to infer and calculate the interaction features of all participants after restoration;

[0019] (6) In the fine-grained reconstruction module and the coarse-grained key figure extraction module, a Maxpool layer is added after each GCN layer that integrates the participant features, and the group features output by multiple maximum pooling layers are added and fused together to form the group-level features of the character interaction; the dynamic and static features in step (4) are also connected to a Maxpool layer to aggregate them into group-level features; the group-level features of the character interaction are added and fused with the group-level features to obtain multi-level character features;

[0020] (7) After transposing the dimensions of the dynamic and static features, the spatial context of all frames is modeled using Sptial-Transformer, and then the temporal context of all frames is modeled using Temporal-Transformer to obtain complex spatiotemporal interaction features;

[0021] (8) Integrate multi-level character features with complex spatiotemporal interaction features to predict the probability of occurrence of each category of group behavior Using dynamic feature X D Predict the probability of each category occurring Add the two predicted probability results and select the highest result P.

[0022] In the present invention, the pretreatment in step (1) includes:

[0023] For the video input S, it is divided into K time blocks of equal time {S1, S2, ..., S k}, the number of video frames in each time block is X t ,t=1,2,…,T, there are N bounding boxes for each frame Get image sequence Where 3 represents the number of channels, T represents the number of frames extracted each time, H and W represent the height and width of the video frame respectively, and the input data is normalized;

[0024] The skeleton node data is obtained by HRNet and is defined as Wherein, N represents N participants in each frame, 3 represents three dimensions of the skeleton node data, respectively, X coordinate, Y coordinate and confidence of the skeleton node, N joint represents N joint skeleton nodes of each participant, N joint =18 is fixed in the experiment, and the input data is normalized.

[0025] In the application, the extraction process of the dynamic characteristics of the participants as a whole is as follows:

[0026] Firstly, the spatial graph G t =(V t ,E t ) of each participant in the t frame is constructed, wherein V t ={v ti |i=1,...,N ti} represents the joint node of the participant, and the edge set E joint ={e t |i,j=1,...,N ij ,i≠j} represents the relationship between the joint nodes v joint and v tj , represents the feature of each node, wherein N joint represents the number of nodes, and C' represents the node attribute dimension; when constructing the adjacency matrix A, it is noted that the movement range of the human body joint is limited, and only the interaction between the node v ti and its neighborhood set N(v ti )={v tj |d(v ti ,v tj )≤1,j≠i} is considered; wherein d(v ti ,v tj ) represents the distance between the joint nodes v ti and v tj ; input into the spatial graph convolution network to obtain the spatial feature X S of the upper joint node in the human body and the updated adjacency matrix A S , and the formula is as follows:

[0027]

[0028] Then, X S is put into the time graph convolution network to capture the spatiotemporal motion feature X ST of the same joint node in different frames, and the time graph G i =(V i ,E i ) is created according to the i-th node in different frames, wherein Vi = {v ti | t = 1,2,...,T}, E i = {e tt′ | t,t' = 1,2,...,T,t'≠t}, Similar to S-GCN, only considering the neighborhood of the joint node, only considering 3 frames around the t frame, the formula is as follows:

[0029]

[0030] Secondly, the spatio-temporal motion feature X ST of the internal joint node of the human body is put into the Temporal Attention module for dynamic feature extraction of the participants as a whole,

[0031] X ST ←Temporal Attention(X ST ) (4).

[0032] In the present application, a plurality of coarse-grained key person extraction modules perform information extraction process on a plurality of key persons:

[0033] Firstly, each participant node in the graph corresponds to a set of groups wherein N is the number of participants, is the dynamic and static feature of the i th participant, that is, the i th participant feature in the dynamic and static feature X ID in step (4); is the position feature of the i th participant represented by the center coordinates; the dynamic and static features and the position features are fused into the participant features input into the GCN; after establishing the initial participant relationship, the GCN is used for further relationship reasoning; the original participant relationship X A is obtained through formula (7) to obtain the participant feature X B ;

[0034]

[0035] wherein, A is the adjacency matrix of the input graph, I is the unit matrix, is the degree matrix of, W is the learned weight matrix, and σ(·) is the activation function RELU;

[0036] Then, in order to pay more attention to the interaction between key participants and extract key participant information as consistency constraint information of multiple interactions into the space-time interaction, the SAGpool method is used to extract coarse-grained key person information; the key of SAGpool lies in that it uses GCN to provide self-attention score Score; Score represents the importance of different participants in each frame, and the calculation formula is as follows:

[0037]

[0038] Wherein, Score is the score obtained by GCN; the top k maximum score nodes are selected to constitute the final person selection mask Score N×1 mask Wherein, The pooling ratio λ is a hyperparameter, which determines the number of nodes to be retained; idx = rank (Score, k), the function rank returns the index of the top K largest scores in Score, idx is the index operation, Score mask = Score idx ;

[0039] Finally, according to the obtained Score mask , a more coarse-grained relationship graph is constructed, the node features X (l) are calculated according to the person selection mask, and the feature matrix X (l+1) and the adjacency matrix A (l+1) are updated, wherein, X idx,: is the feature matrix with row index, X (l+1) = X (l) ⊙ Score mask , A idx,idx is the adjacency matrix with row index and column index, and ⊙ represents the broadcast element product.

[0040] In the present application, three coarse-grained key person extraction modules are stacked in the process of extracting information of multiple key persons, and the key person features

[0041] In the present application, the process of restoring information of all key persons by using multiple fine-grained person information restoration modules is as follows:

[0042] Firstly, the Unpool layer in the fine-grained person information restoration module restores the key nodes to their original positions in the fine-grained relationship graph according to the index recorded by the key person extraction module, so as to restore the fine-grained relationship graph; the GCN layer captures the interaction between the restored nodes;

[0043] ​Then, we add a jump connection between the SAGpool layer of the coarse-grained key person extraction module and the Unpool layer of the fine-grained person information restoration module to better reconstruct the fine-grained person relationship graph and transfer the topological structure and deep features of the relationship graph between the two modules. The specific formula is as follows:

[0044]

[0045] Where idx∈Z *k is the index of k key nodes, X (l) is the feature matrix of the first layer; X (l) The features X obtained by the three key person information extraction modules kv ; Represents the initial feature matrix when the Unpool layer in the fine-grained character information restoration module reconstructs the character relationship, with a value of 0; the function reconstruct will convert X according to the corresponding index recorded in idx (l) The row vectors in are assigned to In, X (l+1) Represents the reconstructed fine-grained character relationship graph, and the row vector of the record index in idx will be X (l) The row vectors in are updated, while the other row vectors remain 0;

[0046] Finally, as the information between the Unpool layers propagates, the last layer restores the feature information of all characters.

[0047] In the present invention, the number of fine-grained character information restoration modules is consistent with the number of coarse-grained key character extraction modules.

[0048] Compared with the prior art, the present invention has the following beneficial effects:

[0049] (1) In the present invention, the dynamic and static features of participants are extracted respectively for the input of multimodal data (skeleton data and RGB data). The dynamic features can characterize the hidden features that are highly relevant to group activities, such as direction and speed. The method of extracting the dynamic features of participants from the local to the overall using the motion trajectory module aims to provide supplementary information for the single static feature, improve the accuracy and stability of model recognition, and combine the static and dynamic features to analyze the group behavior of participants more comprehensively and meticulously.

[0050] (2) In the present application, by extracting coarse-grained key person information and restoring fine-grained person information, the interaction of non-key persons is excluded to prevent interference with the correct action recognition of the model; in the coarse-grained key person information extraction process, the SAGpool method is used to extract coarse-grained key person information, which can flexibly adjust the number of key persons, increase the adaptability and maintainability of the overall model, and solve the problem that the traditional attention mechanism cannot adjust the attention range in time according to the task demand; in the fine-grained person information restoration process, the information of the extracted key persons is reintegrated into the original fine-grained graph, maintaining a comprehensive understanding of the entire scene and utilizing the information of all participants, ensuring the comprehensiveness of the analyzed group behavior and enhancing the generalization ability of the model to different scenes.

[0051] (3) The present application applies Spatial-Transformer and Temporal-Transformer to model time and space interaction respectively, analyzes the time and spatial context of behavior occurrence as supplementary information of the above-mentioned person interaction features, so as to more deeply understand the dynamic changes and evolution process of group behavior; in order to emphasize key person information, the key person information is used as a consistency constraint to guide the attention point of the space-time interaction network, enhancing the recognition ability of the model to the space-time dynamics of key persons. BRIEF DESCRIPTION OF DRAWINGS

[0052] Figure 1 The method framework diagram of the present application.

[0053] Figure 2 The motion trajectory module schematic diagram in the present application.

[0054] Figure 3 The coarse-grained key person extraction schematic diagram in the present application.

[0055] Figure 4 The multi-level person feature extraction schematic diagram in the present application.

[0056] Figure 5 The Spatial Transformer and Temporal Transformer structure schematic diagram.

[0057] Figure 6 The volleyball dataset example diagram in the present application.

[0058] Figure 7 The collective activity dataset example diagram in the present application.

[0059] Figure 8 The motion trajectory module precision influence schematic diagram in the present application.

[0060] Figure 9 The person interaction and space-time interaction confusion matrix in the present application.

[0061] Figure 10 t-SNE classification visualization for different parts of the invention.

[0062] Figure 11 Confusion matrix comparison chart for CAD in the invention. DETAILED DESCRIPTION

[0063] The technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments of the present application.

[0064] In order to identify complex group behaviors in different scenarios, a group behavior recognition method based on dynamic and static features and multiple interaction networks is proposed. Referring to Figures 1-11 , the framework of dynamic and static features and multiple interaction networks (DSMI) will be described in detail. First, the preprocessing of the original input data and the extraction method of dynamic and static features are introduced; then, the description of the character interaction network and the space-time interaction network is expanded, and the selection rules of key characters and the formation of multiple interaction consistency constraints are explained; finally, the multiple interaction features are fused, and the fused features and dynamic features are fused for group behavior recognition.

[0065] Figure 1 The method framework is shown in Figure 1 , where x3 means that the key character extraction module and the Maxpool layer stacked three times, and the detail restoration module and the Maxpool layer stacked three times, for details, see Figure 4 . The framework consists of three parts. The first part is feature extraction, which uses RGB modal data and human skeleton node modal data to input convolutional neural network and motion trajectory module to obtain dynamic and static features of participants; the second part introduces adaptive multiple interaction consistency constraints, which inputs the dynamic and static features of the first part into the multiple interaction network to extract character and space-time interaction features, and inputs the key character information in the character interaction into the space-time interaction network as a consistency constraint, so that the overall network can adaptively adjust the range of key characters, enhance the accuracy and flexibility of the network. The third part introduces the process of feature fusion and classification result level fusion to judge group behavior, which fuses the character interaction features and the space-time interaction features to obtain the classification results of group behavior, and then fuses the classification results obtained by the dynamic features to recognize the group behavior.

[0066] 1. Data preprocessing and dynamic and static feature extraction

[0067] The process of processing the original input data and extracting the dynamic and static features of the participants using convolutional neural network and motion trajectory module will be described.

[0068] 1.1 Data Preprocessing

[0069] In order to better put the original data into the deep network to extract features, the original video input and skeleton node input need to be preprocessed first.

[0070] For the video input S, it is divided into K time blocks of equal time {S1, S2, ..., S k}, the default time interval is 10 frames. The number of video frames in each time block is X t ,t=1,2,…,T, there are N bounding boxes for each frame Get image sequence Input into the model DSMI, where 3 represents the number of channels, T represents extracting T frames each time, H and W represent the height and width of the video frame respectively, and the input data is normalized.

[0071] The skeleton node information is obtained by HRNet and is defined as Among them, N represents that there are N participants in each frame, 3 represents that the skeleton node data has three dimensions, namely the X coordinate and Y coordinate of the node and the confidence of the skeleton node. joint Each participant has N joint Skeleton nodes, N is fixed in the experiment joint =18, the input data is normalized.

[0072] 1.2 Extraction of dynamic and static features

[0073] The process of identifying group behavior requires analyzing the movements of each participant, so extracting participant features is essential. Participant features are categorized as static and dynamic. Static features primarily reflect appearance, while dynamic features primarily reflect movement. For example, in volleyball, a winning point often occurs when a player spikes the ball and another player fails to catch it. Analyzing a spike and a missed catch is based on the player's appearance. However, other actions may occur before and after the critical spike and missed catch, requiring a coherent analysis of these movements to ultimately determine group behavior. Therefore, both static and dynamic characteristics of participants are crucial for identifying group behavior.

[0074] In view of the fact that previous methods using optical flow or three-dimensional networks to extract dynamic features of participants are susceptible to interference and highly complex, the dynamic features are extracted by using participant skeleton nodes. The input of skeleton nodes is easy to obtain and is simple and lightweight. After inputting it into graph convolution, accurate and stable participant motion features can be obtained, thereby improving the model's prediction ability.

[0075] (a) Extraction of static features

[0076] First, After Inception-v3, more comprehensive multi-scale features are extracted. Then, RoIAlign is used to extract the static features of all participants in each frame to obtain X I ∈R T×N×C×K×K , where C represents the number of channels, N represents the number of participants in each frame, and K×K is the size of the feature map for each participant.

[0077]

[0078] (b) Extraction of dynamic features

[0079] like Figure 2 As shown in the figure, the motion trajectory module consists of S-GCN, T-GCN and TemporalAttention. S-GCN and T-GCN extract local motion information of the participant's internal joints, and TemporalAttention extracts the overall motion information of the participant.

[0080] First, construct the spatial graph G of each participant at frame t t =(V t ,E t ), where V t ={v ti ∣v ti ,i=1,…,N joint} represents the participant joint node, edge set E t ={e ij |i,j=1,...,N joint ,i≠j} represents the joint nodes vti and v tj The relationship between Represents the characteristics of each node, where N joint Represents the number of nodes, C' represents the node attribute dimension. When constructing the adjacency matrix A, it is noted that the range of movement of human joints is limited, and only node v is considered. ti and its neighborhood set N(v ti )={v tj ∣d(v ti ,v tj )≤1,j≠i}. Among them, d(v ti ,v tj ) represents the joint node v ti and v tj The distance between Input into the spatial graph convolutional network (S-GCN) to obtain the spatial features X of the upper joint points inside the human body S And the updated adjacency matrix A S , the formula is as follows:

[0081]

[0082] Then, the X S is put into the time graph convolution network (T-GCN) to capture the spatiotemporal motion features X ST of the same joint in different frames i = (V i , E i ), where V i = {v ti | t = 1, 2, …, T}, E i = {e tt′ | t, t' = 1, 2, …, T, t'≠t}, which is similar to the S-GCN only considering the neighborhood of the joint, only considering 3 frames around the t frame, and the formula is as follows:

[0083]

[0084] Secondly, the spatiotemporal motion features X ST of the internal joints of the human body are put into the Temporal Attention module for extraction of the overall motion information of the participant. The Temporal Attention module can dynamically give different attention weights to the inputs of different frames, so that the model can more focusedly focus on important frames, thereby accurately and effectively extracting the overall motion information of the participant.

[0085] X ST ← Temporal Attention (X ST ) (4)

[0086] Finally, in order to fuse the participant static features extracted by the Inception-v3 with the features, the X ST is changed in dimension, as shown in the following formula:

[0087]

[0088] The static features X I and the dynamic features X D are added to obtain the total features of the participant The formula is as follows:

[0089]

[0090] In general, S-GCN and T-GCN respectively model the spatial features of different skeleton points in the same frame and the temporal features of the same skeleton point in different frames, which belong to the local motion information of the skeleton within the participant, while the overall motion information of the participant is extracted through the Temporal Attention module.

[0091] The first stage aims to extract participant features, including static appearance and dynamic motion characteristics. This feature extraction enables a more comprehensive and detailed analysis of group behavior, providing more complete participant information for the second stage, which models multiple interactions between participants.

[0092] 2. Multiple interaction networks based on consistency constraints

[0093] In group action recognition, the spatiotemporal contexts of some participants are also highly complex and diverse. For example, in volleyball matches, in R-set and L-pass actions, participants collaborate extensively, but the spatiotemporal context information varies little. This means that the amount of information about person interactions is large, but the amount of information about spatiotemporal interactions is small. In R-spike and R-winpoint actions, the interactions between participants are not significant over a certain period of time, but the spatiotemporal context information significantly assists in the recognition of participant actions. Therefore, independently and clearly modeling the person interactions and spatiotemporal interactions allows the model to more comprehensively capture the participant behaviors and improve recognition accuracy. Interactions with non-critical individuals may interfere with the model's ability to correctly identify correct actions. Therefore, the model needs to prioritize interactions with critical individuals and suppress those with non-critical individuals. Key person information extracted from person interactions is input into the spatiotemporal interactions as a consistency constraint, enabling the overall model to adaptively adjust the scope of key individuals based on the task.

[0094] To address the issues of incomplete interaction features and unequal interaction relationships between participants, we propose a multi-interaction network based on consistency constraints. This independently models participant interactions and spatiotemporal interactions, and imposes consistency constraints between the two. This allows the overall model to adaptively adjust the scope of key actors, more effectively adapting to and optimizing the execution of downstream tasks. This will be explained below.

[0095] like Figure 3 As shown, Figure 3 (a) Schematic diagram of coarse-grained key figure extraction. Figure 3(b) represents the fine-grained key-actor restoration sketch. The coarse-grained graph is restored to the original position, and the skip connection is added when restoring the key-actor information in the inverse process, which can transfer the topological structure and deep features of the relationship graph from the coarse-grained key-actor extraction module to the fine-grained key-actor information restoration module. The Key-actor Graph* number in the figure represents the key-actor graph obtained after GCN inference of the yellow Key-actor Graph.

[0096] 2.1, Key-actor Interaction

[0097] (a) Coarse-grained key-actor extraction

[0098] In the human visual perception system, some areas in the image have a decisive influence on the decision-making process. For example, in a volleyball match, when a player performs a spike action, the line of sight will often focus on the player performing the key action, rather than other defenders on the court. These players performing the main action are called key actors, and their interactions are crucial to understanding the behavior of the entire group. The interactions between non-key actors are less important and may even interfere with the final prediction results. Therefore, the interactions of key actors need to be highlighted and the interactions of non-key actors need to be suppressed in the analysis process.

[0099] To solve the problem that the traditional attention mechanism cannot adjust the focus range in a timely manner according to the task requirements, the SAGpool method is used to extract coarse-grained key-actor information, which can flexibly adjust the number of key actors and increase the adaptability and maintainability of the overall model.

[0100] First, when the distance between participants is closer, the connection and mutual influence between them is more significant, which helps to more effectively perform relationship inference. Therefore, each participant node in the graph is mapped to a set of sets where N is the number of participants, is the dynamic and static features of the ith participant, i.e., X ID is the ith participant feature in the first stage. is the position feature of the ith participant represented by the center coordinates. The dynamic and static features and the position features are fused into the participant features for the final GCN input. After establishing the initial participant relationship, GCN is used for further relationship inference. The original participant relationship X A is obtained by formula (7) to obtain the participant feature X B .

[0101]

[0102] where, A is the adjacency matrix of the input graph, I is the identity matrix, is where D is the degree matrix, W is the learned weight matrix, and σ(·) is the activation function RELU.

[0103] Then, to pay more attention to the interaction between key participants and extract key participant information as consistency constraint information of multiple interactions into the spatio-temporal interaction, the SAGpool method is used to extract coarse-grained key person information. The key of SAGpool lies in that it uses GCN to provide self-attention scores Score. Score represents the importance of different participants in each frame, and the calculation formula is as follows:

[0104]

[0105] where Score ∈ R N×1 is the score obtained by GCN. The top k largest score nodes are selected to form the final person selection mask Score mask where, The pooling ratio λ ∈ (0, 1] is a hyperparameter that determines the number of nodes to be retained. idx = rank(Score, k), the function rank returns the index of the top K largest scores in Score, idx is the index operation, Score mask = Score idx .

[0106] Finally, according to the obtained Score mask , a more coarse-grained relationship graph is constructed, and the node features X (l) are calculated according to the person selection mask, and the feature matrix X (l+1) and the adjacency matrix A (l+1) are updated, where, X idx,: is the feature matrix with row index, X (l+1) = X (l) ⊙ Score mask , A idx,idx is the adjacency matrix with row and column index, and ⊙ represents the broadcast element product.

[0107] With the propagation of information between SAGpool layers, the last layer obtains the most critical person feature information. Three key person information extraction modules are stacked in total, and the key person feature obtained by the last module will be transmitted into the spatio-temporal interaction network as an external key person consistency constraint. The process is shown in Figure 3 (a).

[0108] (b) fine-grained person information restoration

[0109] In order to maintain a comprehensive understanding of the entire scenario and utilize the information of all participants, it is necessary to reintegrate the information of key figures into the original fine-grained graph. This ensures the comprehensiveness of the analyzed group behavior and enhances the model's generalization ability to different scenarios.

[0110] Fine-grained character information restoration is the inverse operation of coarse-grained key character extraction, aiming to reconstruct the fine-grained relationship graph. Figure 3 As shown in (b), first, the Unpool layer in the fine-grained reconstruction module puts the key nodes back to their original positions in the fine-grained relationship graph according to the indices recorded by the key person extraction module, thereby restoring the fine-grained relationship graph. The GCN layer captures the interactions between the restored nodes.

[0111] Then, adding a jump connection between the SAGpool layer of the coarse-grained key person extraction module and the Unpool layer of the fine-grained person information restoration module can better reconstruct the fine-grained person relationship graph. It can convey the topological structure and deep features of the relationship graph between the two modules. The specific formula is expressed as follows:

[0112]

[0113] Where idx∈Z *k is the index of k key nodes, X (l) is the feature matrix of the first layer. (l) The features X obtained by the three key person information extraction modules kv . Represents the initial feature matrix when the Unpool layer in the fine-grained character information restoration module reconstructs the character relationship, and its value is 0. The function reconstruct will convert X to the corresponding index recorded in idx. (l) The row vectors in are assigned to In, X (l+1) Represents the reconstructed fine-grained character relationship graph, and the row vector of the record index in idx will be X (l) The row vectors in are updated, while the other row vectors remain 0.

[0114] Finally, as the information between the Unpool layers propagates, the last layer restores the feature information of all characters. It should be noted that the features X of all characters derived and restored by GCN are C With the original X B Although the shape is the same, X C This will highlight the characteristics of key figures and more accurately represent group behavior. The number of blocks in the fine-grained figure restoration module and the key figure information extraction module remains the same, at 3.

[0115] After adding a GCN layer behind the three fine-grained character information restoration modules, the purpose is to infer the interaction features of the restored N participants. C After the last GCN, X' is obtained D .

[0116] (c) Extraction of multi-level character features

[0117] Group behavior is composed of multiple participant behaviors, while participant behavior is guided by group behavior. Focusing only on participant-level features, the model is easily limited to specific individuals and difficult to generalize to new groups and scenarios. In order to facilitate multi-level analysis and meet the needs of different tasks, two levels of features are extracted, namely the fine-grained key character extraction and the fine-grained character information restoration, which obtain the interaction information at the participant level. By aggregating the participant-level features through the max-pooling layer, the group-level features can be obtained.

[0118] First, in the fine-grained reconstruction module and the coarse-grained key character extraction module, a Maxpool layer is added after each GCN layer that fuses participant features. The purpose is to effectively aggregate the participant-level features to generate more comprehensive group-level features.

[0119] Then, the group features output by multiple max-pooling layers are added and fused together to form the participant interaction group-level features The participant dynamic and static features X extracted in the first stage ID are also connected to a Maxpool layer to aggregate them into group-level features. Then, are added and fused together with X ID .

[0120]

[0121] Finally, the above interaction between participants is performed on the same frame, so the participant interaction features on T frames can be represented as The process diagram is as follows Figure 4 .

[0122] The character interaction module extracts coarse-grained key character information through the SAGpool method and restores it to the fine-grained participant graph using inverse operations to fully capture the interaction features in group behavior. At the same time, the key character information is used as a consistency constraint to guide the focus of the spatio-temporal interaction network, enhancing the model's ability to recognize key character spatio-temporal dynamics. The invention extracts multi-level interaction features from the participant level and the group level, which not only improves the model's generalization ability, but also ensures the performance stability in new scenarios, providing a more in-depth and comprehensive perspective for group behavior recognition.

[0123] 2.2, Space-time interaction

[0124] The space-time context of participants in group behavior has an important influence on their behavior judgment, and the space-time context information is crucial for a comprehensive understanding of group behavior. Spatial-Transformer and Temporal-Transformer are applied to model time and space interaction respectively, and the time and space context of behavior occurrence is analyzed as supplementary information of the above-mentioned character interaction features, so as to more deeply understand the dynamic changes and evolution process of group behavior.

[0125] (a) Spatial interaction

[0126] Firstly, the participant features output in the first stage are transposed to obtain:

[0127]

[0128] Then, when modeling spatial interaction, the time dimension T is regarded as a batch dimension, the spatial interaction of N characters in the same frame is considered, and Spatial-Transformer is applied to model the spatial context of all frames. The participant features X ID in the t-th frame are input into the participant query Q (t) , the key character features X kv extracted by the character interaction module are mapped into the key K (t) and the value V (t) , which are transmitted into Spatial-Transformer as the consistency constraints of multiple interactions, so that the network can pay more attention to the spatial interaction information of key characters. Wherein, is mapped through the weight matrix , which are all learnable parameters with the shape of D e ×D e . The similarity between the participant features and all key character features is calculated to generate a similarity matrix.

[0129] Next, in order to make the gradient update more stable, divide each element of the similarity matrix by the dimension size of Q (t) and K (t) , and normalize the result by using the Softmax function.

[0130] Finally, after a MatMul operation between the normalized similarity matrix and V (t) , a residual connection is made with V (t) to obtain the attention output V' (t) . The residual connection can retain the key character information of the original input and improve the expression ability of the model. The calculation formula is as follows: ​

[0131]

[0132] Output of attention V' (t) Input the feedforward neural network (FFN) to get the spatial feature V (t) Stack all frame feature maps {V (t) The shape becomes Get the final spatial interaction feature The structural diagram is as Figure 5 (a).

[0133] (b) Time interaction

[0134] After modeling the spatial interaction of group behavior by Spatial-Transformer, the same operation method is used to model the time interaction. The difference between Temporal-Transformer and Spatial-Transformer is that it considers the spatial dimension as the batch dimension, that is, it considers the time interaction of the same person in different frames. The Temporal-Transformer is applied to model the time context of all frames. The structural diagram of Temporal-Transformer is as Figure 5 (b).

[0135] First, the output time dynamic feature of the nth individual is represented as V Stack the generated feature maps {V (n) The shape becomes

[0136] Then, in order to fuse the features with the person interaction, V t is dimensionally converted:

[0137]

[0138] Finally, the spatiotemporal interaction features of N participants are aggregated into group-level features by a max-pooling layer, and the formula is:

[0139]

[0140] The above multi-interaction modeling stage is the second stage of the overall model, and multi-level person interaction features and complex spatiotemporal interaction features are obtained. Adding consistency constraints in person interaction and spatiotemporal interaction makes the second stage focus on the interaction information of key persons together. Independently and clearly modeling person and spatiotemporal interaction can make the model learn more comprehensive interaction details to better adapt to complex crowds and different scenes.

[0141] 3. Integrated fusion process

[0142] First, the character interaction and space-time interaction are fused, and the formula is as follows:

[0143]

[0144] Then, the fused feature X interaction is used to predict the probability of occurrence of each class of group behavior Meanwhile, the dynamic feature X D output by the motion trajectory branch also predicts the probability of occurrence of each class The two predicted probability results are added together, and the highest result P is selected.

[0145]

[0146] Where N c represents the total number of group behavior classes.

[0147] Finally, the class corresponding to P is the final prediction result of the group behavior.

[0148] 4. Method evaluation

[0149] In order to evaluate the proposed method, ablation experiments and comparative experiments were conducted on public datasets Volleyball Dataset (VD) and Collective Activity Dataset (CAD) to verify the effectiveness of the group behavior recognition method based on dynamic and static features and multiple interaction reasoning. First, the details of the dataset and the implementation of the method are briefly introduced. Next, the method of the present application is compared with the previous most advanced method. Then, in order to prove the respective contributions of different parts, some ablation studies are conducted. The visualization results and the corresponding analysis are given.

[0150] 4.1) Metrics

[0151] Multi-Class Accuracy (MCA) and Mean Per-Class Accuracy (MPCA) are two commonly used precision measurement standards for evaluating the performance of classification models. MCA is suitable for overall model performance evaluation, while MPCA is more suitable for detailed analysis of each class classification, especially for class imbalance datasets, and has more reference value. Based on the above description, MCA is used to evaluate the performance of Volleyball dataset, and MPCA is used to evaluate the performance of Collective dataset. The accuracy calculated by MCA or MPCA is called precision.

[0152] MCA refers to the average of the classification accuracy of all classes in a multi-class task. It is a simple and intuitive performance indicator, calculated as follows:

[0153]

[0154] where N is the number of classes, TP i is the number of true positives for the i-th class, FP i is the number of false positives for the i-th class.

[0155] MPCA refers to the average of the classification accuracy of each class in a multi-class task. Compared with MCA, MPCA pays more attention to the classification performance of each class, especially suitable for class imbalance situations. The calculation method is as follows:

[0156]

[0157] where Acc i is the classification accuracy of the i-th class.

[0158] 4.2) Dataset Introduction

[0159] The DSMI framework was evaluated on the following two public GAR datasets that provide tracking annotations and bounding boxes.

[0160] Volleyball Dataset (VD): This dataset consists of 4830 video segments of 55 volleyball games (3493 segments for training and 1337 segments for testing), each segment has 41 frames. The middle frame of each segment contains bounding box coordinates, single action labels and group activity labels. Single action labels contain 9 actions: setting, digging, falling, jumping, blocking, moving, spiking, waiting and standing. Group activity labels contain 8 activities, i.e., right setting, right passing, right spiking, right winpoint, left setting, left passing, left spiking and left winpoint. As shown in Table 1, to make a fair comparison with the GAR method, the coordinates of each participant and the training / test split provided in [1, 2, 4, 6] are followed. Figure 6

[0161] Collective Activity Dataset (CAD): This dataset consists of 44 video segments captured by a low-resolution handheld camera with dynamic viewpoints, with frame numbers ranging from 194 to 1814. The dataset contains five different collective activity labels, i.e., crossing, waiting, queueing, walking, talking, six different individual action labels, i.e., NA, crossing, waiting, queueing, walking, talking and eight different individual pose labels (not used in the work). The middle frame of every ten frames contains bounding box coordinate annotations and individual action labels, and the actions of most individuals in the same scene determine the group activity. Following the equivalent experimental setup in [1], 2 / 3 of the dataset is selected for training and the rest for testing. As shown in Table 2, following the methods of [5, 6, 8], the crossing class and the walking class are merged into the moving class. Figure 7

[0162] 4.3) Experimental details

[0163] ​​The Inception-v3 model pre-trained on ImageNet is selected as the feature extraction network, following the strategy widely used in GAR. For VD and CAD, T = 3 frames are sampled to input the model when training the model, and T = 9 frames are sampled when testing the model. When extracting static features, the cropping size of RoIAlign is 5x5, and the dimension of actor features is 1024. When extracting dynamic features, the input and output channels of S-GCN and T-GCN are set to 3 and 256 respectively, and the number of attention heads in the temporal attention module is set to 8. In order to make a fair comparison with other papers, the maximum number of actors N in VD is set to 12, and the maximum number of actors N in CAD is set to 13. For VD, the initial learning rate (lr) is set to 2e-4. The lr decreases with the increase of epoch, and is reduced to 1e-4, 5e-5 and 1e-5 at the 40th, 80th and 120th epoch respectively. For CAD, the lr starts from 1e-4 and remains unchanged. On both datasets, the optimizer of the model is ADAM, the loss function is cross-entropy function, and the evaluation criteria for VD is MCA, and the evaluation criteria for CAD is MPCA. ARG is selected as the baseline method. The environment and configuration of the experiment are shown in Table 1.

[0164] Table 1 Experimental environment

[0165]

[0166] 4.4) Ablation experiments

[0167] In order to prove the respective contributions of different parts to the method, some ablation studies are carried out, using MCA and MPCA as experimental evaluation indicators on VD and MPCA as experimental evaluation indicators on CAD, and the specific methods are as follows:

[0168] Comparison of the composition and accuracy of different baselines:

[0169] B1-Baseline Model: This baseline model includes Inception-v3 network, RoIAlign, embedding layer and Softmax classification layer.

[0170] B2-Baseline Model+Motion: This baseline is composed of the baseline model and the motion trajectory module.

[0171] B3-Baseline Model+Motion+Actor Interaction: This baseline is composed of the baseline model, the motion trajectory module and the actor interaction module.

[0172] B4-Baseline Model+Motion+S-T Interaction: This baseline consists of the base model, the motion trajectory module and the spatio-temporal interaction module.

[0173] Ours method: The first stage of the complete model consists of the Inception-v3 network and the motion trajectory module, the second stage contains the person interaction and the spatio-temporal interaction module, and the key person information extracted by the person interaction module is transmitted to the spatio-temporal interaction module as external constraints, and finally the classification results of the dynamic features extracted by the motion trajectory module and the classification results of the multiple interaction features are fused to judge the group behavior.

[0174] Table 2 Accuracy comparison of each part of the ablation experiment

[0175]

[0176] Note: "Motion" in the table represents the motion trajectory module

[0177] In addition to the model structure, all training settings are the same, as shown in the table, the complete model is better than these baselines. The following analyzes these comparisons:

[0178] As shown in Table 2. The comparison between baseline B2 and B1 confirms the effectiveness of the motion trajectory module in the feature extraction stage. The MCA of baseline B2 is improved by 4.0% and 2.5% on VD and CAD respectively compared with baseline B1, which proves that the Motion Trajectory Module can supplement the dynamic motion information for GAR. The comparison between baseline B3 and baseline B2 confirms the effectiveness of the person interaction module. The MCA of baseline B3 is improved by 0.7% on VD and 1.2% on CAD compared with baseline B2. It proves that the person interaction information helps the model to identify the unequal interaction relationship between participants and thus distinguish similar behaviors. The comparison between baseline B4 and baseline B2 confirms the effectiveness of the spatio-temporal interaction module. The MCA of baseline B4 is improved by 0.6% on VD and 0.4% on CAD compared with baseline B2. It proves that the participant context information extracted by this module helps the model to distinguish group behaviors in complex environments. The comparison between the overall model and baseline B3, B4 confirms the effectiveness of multiple interactions. The recognition accuracy of the overall model is higher, which shows that the comprehensive and complete interaction can enhance the adaptability of the model.

[0179] Effectiveness analysis of the motion trajectory module: The accuracy is compared on VD and CAD to prove the effectiveness of the motion trajectory module to other modules, as shown in Figure 8 , as shown in Figure 8 (a), Figure 8(b) It can be seen that the motion trajectory module can help the model to extract the motion hidden information of the participants as complementary information of the static features, and a higher recognition accuracy can be obtained when only using the static and dynamic features of the participants to judge the group behavior. Figure 8 (c), Figure 8 (d) It can be seen that, on the one hand, the dynamic feature inputting the multiple interaction network is beneficial to the model to extract more interaction details with the help of the participant motion information; on the other hand, the fusion of the dynamic feature classification result and the multiple interaction feature classification result significantly improves the accuracy of group behavior recognition (the accuracy is shown in Table 2), which shows that the dynamic feature can capture effective key motion details, thereby optimizing the classification result.

[0180] Multiple interaction effectiveness analysis: The confusion matrix of baseline B3 and baseline B4 on VD proves that the character interaction and space-time interaction network can effectively capture the interaction features and improve the recognition accuracy of the model. As shown in Figure 9 , Figure 9 (a) is the confusion matrix of the character interaction, Figure 9 (b) is the confusion matrix of the space-time interaction. "L" and "R" are abbreviations of "left" and "right" in the group behavior label. On the R_winpoint and L_set two classes of behaviors of the VD dataset, the accuracy of the character interaction and the space-time interaction is flat, and the judgment on the other classes forms a complementary advantage. The character interaction can capture the unequal interaction between characters, and the space-time interaction focuses on capturing the interaction of the participants with the complex space-time. The multiple interaction makes the interaction captured by the model more detailed and complete, so that the group behavior can still be accurately recognized even in an environment with complex space-time background or chaotic crowd, thereby improving the adaptability and stability of the model.

[0181] Classification of different parts can be visualized: Figure 10 which clearly shows the classification ability of the model after adding the character and space-time interaction on VD. Figure 10 (a) shows the classification of baseline B2. It can be seen that the classification boundaries of R_spike class, R_pass class and R_set class are not clear, and some L_pass classes are misclassified as L_spike classes. Figure 10 (b) shows the classification of baseline B3. It can be seen that the classification boundaries of R_set class and R_pass class are not clear, and compared with baseline B2, the R_spike classification is more clear. It can be seen that after adding the character interaction information, the model can learn more differences between the behaviors of the participants. Figure 10 (c) shows the classification of baseline B4. It can be seen that the classification degree and Figure 10 (b) are similar. Figure 10(d) The classification of the method is shown. It can be observed that the classification boundary of the L_set class and the R_winpoint class is more clear than all the previous baselines. From the action itself, R_set is a pad ball action and R_pass is a pass ball action, both of which are completed by a key actor and have high action similarity. Therefore, the classification boundary of the R_set class and the R_pass class is not very clear under different baselines. In this case, the overall model makes the inter-class distance of different classes larger and the intra-class distance of the same class smaller, which indicates that the overall model is more stable and has stronger classification ability than other baseline models.

[0182] Analysis of consistency constraints: The ablation experiments on VD and CAD verified the effectiveness of the key person information extracted in the human interaction as a consistency constraint. The results are shown in Table 3. When the query (Q), key (K) and value (V) are homologous, that is, they all come from the dynamic and static features of the participants in the first stage, the performance of the model is lower than the method of passing the key person information as the key and value into the Spatial-transformer. Specifically, on VD, using the method of Q=K=V=X ID , the MCA and MPCA are 93.1% and 93.4% respectively, while the method of passing the key person information as the key and value into the Spatial-transformer improves the accuracy to 93.8% and 94.1% respectively. It shows that the key person information has a significant advantage for the model to capture the unequal interaction relationship between persons and the complex spatio-temporal dynamics, and the key person who maintains the consistency of multiple interaction attention will improve the stability of the model. When using the method of Q=K=X ID , V=X kv , the method of restricting the consistency of multiple interactions has improved the accuracy compared with the method without using the consistency constraint, but the performance is not as good as the second experimental setting. This is because Q captures the dynamic and static features of the participants, while K and V are derived from the feature information of the key person. This design enables the model to more effectively capture the rich spatio-temporal context features of the key person. When K and V share the same information source, the model can more accurately receive effective constraint information of the key person, thereby achieving better performance.

[0183] Table 3 Comparison of key person information transmission methods

[0184]

[0185] Key person extraction analysis: The present application performs an ablation experiment on the VD to extract the number of key person modules in the person interaction network, as shown in Table 4, the accuracy of using 3 modules is the highest compared with using 1, 2 and 4 key person modules, the MCA is 0.7% higher than using 2 modules, and 0.6% higher than using four. The reason is that too few modules will not be able to filter out the most critical person, the influence of non-key person is still great, and it will interfere with the recognition of group behavior; while too many modules will cause over-filtering of key persons, so that some key persons are not noticed, which is not conducive to the model to reason and recognize the correct behavior. The key person module is composed of a self-attention pooling layer and a graph convolution layer, by stacking different numbers of self-attention pooling layers in the person interaction network to dynamically adjust the attention range, and according to the specific task requirements, the most suitable number of key persons is flexibly selected. In this way, the flexibility and adaptability of the model can be enhanced, and the optimal key person recognition effect can be realized in complex tasks.

[0186] Table 4 Comparison of key person module block number at coarse granularity

[0187]

[0188] Analysis of comprehensive fusion mode: The present application performs an ablation experiment on the VD to verify the effectiveness of the feature level fusion of multiple interactions and the result level fusion of multiple interaction classification results and dynamic feature classification results. The experimental results are shown in Table 5, the feature fusion adopts three typical ways: Addition, Concat, and Gating. The MCA of using the Addition method to fuse multiple interaction features is 1.3% and 0.8% higher than using the Concat method and the Gating method. In addition, the fusion at the result level also enhances the classification performance of the model. This is because the comprehensive fusion strategy can fully utilize the complementarity of different features, and allows the model to adjust according to the importance of the features, increasing the flexibility of the model.

[0189] Table 5 Comparison of feature fusion modes

[0190]

[0191] 4.5) Comparison with previous methods

[0192] In this section, the proposed DSMI method is compared with previous methods on the two public GAR datasets mentioned above.

[0193] (a) Volleyball dataset results: The experimental results are shown in Table 6. The columns of "MCA-VD" and "MPCA-VD" compare the method of the present application with the latest method of VD. It can be observed that the method of the present application improves MCA by 26.9% and MPCA by 26.5% compared with the worst SBGAR [9] method. StagNet [3] understands group activities in videos by combining spatio-temporal attention mechanisms and semantic graph modeling. Compared with previous methods using only RNN and LSTM, StagNet achieves significant improvement. DIN [7] consists of a dynamic relationship (DR) module and a dynamic walking (DW) module. Compared with RNN-based models [9-10], DIN not only spends less computational overhead, but also has better performance, mainly because it can better represent spatio-temporal interaction joint modeling; DIN is also superior to GNN-based methods [2, 4, 11], mainly because of the dynamic nature of the DR and DW modules. The method of the present application (MCA-93.8%) has a significant improvement compared with the method using Inception-v3. Compared with the method [5] using additional optical flow input, the method of the present application uses human skeleton node data to capture the motion of participants. This approach does not require additional calculation of optical flow, and has lower computational complexity. At the same time, the backbone network used is a two-dimensional network, which has much lower computational complexity than using an I3D network.

[0194] Table 6 Precision comparison of the method of the present application and some classic GAR methods on VD

[0195]

[0196] Note: Bold represents the highest precision in the table, and "-" represents that the MPCA of the method is not disclosed.

[0197] (b) Collective activity dataset results: The experimental results are shown in Table 7. Specifically, on the collective activity dataset, the method of the present application achieves SOTA accuracy of 97.4%. Compared with non-deep methods, the accuracy of the method of the present application is much higher; compared with previous studies TCE+STBiP and Dual-AI using Inception-v3 as the backbone network and adopting the same dataset configuration (i.e., merging the categories crossing and walking into moving), the method of the present application achieves a significant improvement of 2.3% and 0.9% in accuracy, respectively. In order to further prove the effectiveness of the method of the present application, the classification results are compared with DIN Figure 11 (a) and MLST Figure 11 (b) method results, as shown in Figure 11 Although the method of the present application Figure 11(c) There are cases that moving class errors are identified as waiting class, which is probably due to the similarity of the actions of the two classes when crossing the road. The method of the present application shows a significant performance improvement in identifying the waiting class, which is mainly due to the rich context information provided by the space-time interaction in multiple interactions. In addition, the method of the present application achieves 100% accuracy on the queueing class and the talking class, further proving the effectiveness of the DSMI framework.

[0198] Table 7 Precision comparison of the method of the present application and some classic GAR methods on CAD

[0199]

[0200] Note: “*” represents the method of merging “Crossing” class and “Walking” class into “Moving” class in CAD.

[0201] The above is only the preferred embodiment of the present application, but the protection scope of the present application is not limited to this. Any person skilled in the art can make equivalent replacements or changes to the technical solutions and inventive concepts of the present application within the technical scope disclosed by the present application, which should be covered within the protection scope of the present application.

[0202] References:

[0203] [1] IBRAHIM M S, MURALIDHARAN S, DENG Z, et al. A hierarchical deep temporal model for group activity recognition [C] / / Proceedings of the IEEE conference on computer vision and pattern recognition, 2016: 1971-1980.

[0204] [2] WU J, WANG L, WANG L, et al. Learning actor relation graphs for group activity recognition [C] / / Proceedings of the IEEE / CVF Conference on computer vision and pattern recognition, 2019: 9964-9974.

[0205] [3] QI M, WANG Y, QIN J, et al. StagNet: An attentive semantic RNN for group activity and individual action recognition [J]. IEEE Transactions on Circuits and Systems for Video Technology, 2019, 30(2): 549-565.

[0206] [4] GAVRILYUK K, SANFORD R, JAVAN M, et al. Actor-transformers for group activity recognition [C] / / Proceedings of the IEEE / CVF conference on computer vision and pattern recognition, 2020: 839-848.

[0207] [5] YAN R, TANG J, SHU X, et al. Participation-contributed temporal dynamic model for group activity recognition [C] / / Proceedings of the 26th ACM international conference on Multimedia, 2018: 1292-1300.

[0208] [6] HAN M, ZHANG D J, WANG Y, et al. Dual-AI: Dual-path actor interaction learning for group activity recognition [C] / / Proceedings of the IEEE / CVF conference on computer vision and pattern recognition, 2022: 2990-2999.

[0209] [7] YUAN H, NI D, WANG M. Spatio-temporal dynamic inference network for group activity recognition [C] / / Proceedings of the IEEE / CVF International Conference on Computer Vision, 2021: 7476-7485.

[0210] [8] YUAN H, NI D. Learning visual context for group activity recognition [C] / / Proceedings of the AAAI Conference on Artificial Intelligence, 2021: 3261-3269.

[0211] [9] LI X, CHOO CHUAH M. Sbgar: Semantics based group activity recognition [C] / / Proceedings of the IEEE international conference on computer vision, 2017: 2876-2885.

[0212]

[10] YAN R, SHU X, YUAN C, et al. Position-aware participation-contributed temporal dynamic model for group activity recognition [J]. 2021, 33(12): 7574-7588.

[0213]

[11] YAN R, XIE L, TANG J, et al. HiGCIN: Hierarchical graph-based cross inference network for group activity recognition [J]. IEEE transactions on pattern analysis and machine intelligence, 2020, 45(6): 6955-6968.

[0214]

[12] LU L, LU Y, WANG S. Learning multi-level interaction relations and feature representations for group activity recognition [C] / / MultiMedia Modeling: 27th International Conference, MMM 2021, Prague, Czech Republic, June 22-24, 2021, Proceedings, Part I 27, 2021: 617-628.

[0215]

[13] AZAR S M, ATIGH M G, NICKABADI A, et al. Convolutional relational machine for group activity recognition [C] / / Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition, 2019: 7892-7901.

[0216]

[14] HU G, CUI B, HE Y, et al. Progressive relation learning for group activity recognition [C] / / Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition, 2020: 980-989.

[0217]

[15] ZHAI X, HU Z, YANG D, et al. Spatial temporal network for image and skeleton based group activity recognition [C] / / Proceedings of the Asian Conference on Computer Vision, 2022: 20-38.

[0218]

[16] ZHU X, ZHOU Y, WANG D, et al. MLST-Former: Multi-level Spatial-Temporal Transformer for Group Activity Recognition [J]. IEEE Transactions on Circuits and Systems for Video Technology, 2022, 33(7): 3383-3397.

Claims

1. A group behavior recognition method based on dynamic and static features and multiple interaction networks, characterized by: The following steps are involved: (1) Preprocessing the original video data and skeleton node data; (2) Inception-v3 is used to extract multi-scale features from the preprocessed video data, and then RoIAlign is used to extract the static features of all participants in each frame; (3) The preprocessed skeleton node data is sequentially input into the spatial graph convolutional network and the temporal graph convolutional network to obtain the spatiotemporal motion features of the internal joints of the human body. The spatiotemporal motion features of the internal joints of the human body are then input into the TemporalAttention module to extract the overall dynamic features of the participant; (4) After changing the dimension of the dynamic features of the whole participant so that their dimension is the same as the static feature dimension, the static feature is fused with the dynamic features of the whole participant after the dimension change to obtain the dynamic and static features; (5) Use multiple coarse-grained key person extraction modules to extract information from multiple key persons, and then use multiple fine-grained person information restoration modules to restore information for all key persons. After the multiple fine-grained person information restoration modules, add a GCN layer to infer and calculate the interaction features of all participants after restoration; (6) In the fine-grained reconstruction module and the coarse-grained key figure extraction module, a Maxpool layer is added after each GCN layer that integrates the participant features, and the group features output by multiple maximum pooling layers are added and fused together to form the group-level features of the character interaction; the dynamic and static features in step (4) are also connected to a Maxpool layer to aggregate them into group-level features; the group-level features of the character interaction are added and fused with the group-level features to obtain multi-level character features; (7) After transposing the dimensions of the dynamic and static features, the spatial context of all frames is modeled using Sptial-Transformer, and then the temporal context of all frames is modeled using Temporal-Transformer to obtain complex spatiotemporal interaction features; (8) Integrate multi-level character features with complex spatiotemporal interaction features to predict the probability of occurrence of each category of group behavior Using dynamic feature X D Predict the probability of each category occurring Add the two predicted probability results and select the highest result P.

2. The method for identifying group behavior based on dynamic and static features and multiple interactive networks according to claim 1 is characterized in that: The preprocessing in step (1) includes: For the video input S, it is divided into K time blocks of equal time {S1, S2, ..., S k }, the number of video frames in each time block is X t ,t=1,2,…,T, there are N bounding boxes for each frame Get image sequence Where 3 represents the number of channels, T represents the number of frames extracted each time, H and W represent the height and width of the video frame respectively, and the input data is normalized; The skeleton node data is obtained by HRNet and is defined as Among them, N represents that there are N participants in each frame, 3 represents that the skeleton node data has three dimensions, namely the X coordinate and Y coordinate of the node and the confidence of the skeleton node. joint Each participant has N joint A skeleton node is used to normalize the input data.

3. The method for identifying group behavior based on dynamic and static features and multiple interactive networks according to claim 1 is characterized in that: The process of extracting the dynamic features of the entire participant is as follows: First, construct the spatial graph G of each participant at frame t t =(V t ,E t ), where V t ={v ti ∣v ti ,i=1,...,N joint } represents the participant joint node, the edge set E t ={e ij |i,j=1,...,N joint ,i≠j} represents the joint node v ti and v tj The relationship between Represents the characteristics of each node, where N joint Represents the number of nodes, C' represents the node attribute dimension. When constructing the adjacency matrix A, it is noted that the range of movement of human joints is limited, and only node v is considered. ti and its neighborhood set N(v ti )={v tj ∣d(v ti ,v tj )≤1,j≠i}; where d(v ti ,v tj ) represents the joint node v ti and v tj The distance between Input into the spatial graph convolutional network to obtain the spatial features X of the upper joint points inside the human body S And the updated adjacency matrix A S , the formula is as follows: Then, X S Put it into the time graph convolution network to capture the spatiotemporal motion features X of the same joint point in different frames ST , create a time graph G based on the i-th node in different frames i =(V i ,E i ), where V i ={v ti |t=1,2,...,T},E i ={e tt′ |t,t′=1,2,…,T,t′≠t}, similar to S-GCN which only considers the neighborhood of the joint point, only the three frames around the t-th frame are considered. The formula is as follows: Secondly, the spatiotemporal motion characteristics X of the joints inside the human body are ST Put it into the Temporal Attention module to extract the dynamic features of the whole participant. X ST ←Temporal Attention(X ST )(4)。 4. The method for identifying group behavior based on dynamic and static features and multiple interactive networks according to claim 1 is characterized in that: Multiple coarse-grained key person extraction modules perform information extraction on multiple key persons: First, each participant node in the graph corresponds to a set of Where N is the number of participants, is the dynamic and static features of the i-th participant, i.e., the dynamic and static features X in step (4) ID The i-th participant characteristics in ; The position feature of the i-th participant is represented by the center coordinates; the dynamic and static features and the position features are fused into the participant features that are finally input into GCN; after establishing the initial participant relationship, GCN is used to further reason about it. The original participant relationship X A The participant characteristics X are obtained through formula (7): B ; in, A is the adjacency matrix of the input graph, I is the identity matrix, yes The degree matrix, W is the learned weight matrix, and σ(·) is the activation function RELU; Then, in order to pay more attention to the interactions between key participants and extract key participant information as consistency constraint information for multiple interactions and pass it into the spatiotemporal interaction, the SAGpool method is used to extract coarse-grained key person information; the key to SAGpool is that it uses GCN to provide a self-attention score Score; Score represents the importance of different participants in each frame, and the calculation formula is as follows: Among them, Score∈R N×1 is the score obtained by GCN; select the top k largest scoring nodes to form the final character selection mask Score mask ,in, The pooling ratio λ∈(0,1] is a hyperparameter that determines the number of nodes to be retained; idx=rank(Score,k), the function rank returns the index of the top K largest scores in Score, idx It is an index operation, Score mask =Score idx ; Finally, according to the obtained Score mask , build a coarser-grained relationship graph, and calculate the node feature X according to the character selection mask (l) And update the feature matrix X (l+1) and the adjacency matrix A (l+1) ,in, X idx,: is the row-wise indexed feature matrix, X (l+1) =X (l) ⊙Score mask , A idx,idx is the adjacency matrix of row and column indices, and ⊙ represents the broadcast element-wise product.

5. The method for identifying group behavior based on dynamic and static features and multiple interactive networks according to claim 4 is characterized in that: In the process of extracting information from multiple key figures, a total of three coarse-grained key figure extraction modules are stacked, and the key figure features obtained by the last module are 6. The method for identifying group behavior based on dynamic and static features and multiple interactive networks according to claim 1, characterized in that: The process of restoring information of all key figures using multiple fine-grained character information restoration modules is as follows: First, the Unpool layer in the fine-grained person information restoration module places key nodes back to their original positions in the fine-grained relationship graph based on the indexes recorded by the key person extraction module, thereby restoring the fine-grained relationship graph; the GCN layer captures the interactions between the restored nodes; Then, we add a jump connection between the SAGpool layer of the coarse-grained key person extraction module and the Unpool layer of the fine-grained person information restoration module to better reconstruct the fine-grained person relationship graph and transfer the topological structure and deep features of the relationship graph between the two modules. The specific formula is as follows: Where idx∈Z *k is the index of k key nodes, X (l) is the feature matrix of the first layer; X (l) The features X obtained by the three key person information extraction modules kv ; Represents the initial feature matrix when the Unpool layer in the fine-grained character information restoration module reconstructs the character relationship, with a value of 0; the function reconstruct will convert X according to the corresponding index recorded in idx (l) The row vectors in are assigned to In, X (l+1) Represents the reconstructed fine-grained character relationship graph, and the row vector of the record index in idx will be X (l) The row vectors in are updated, while the other row vectors remain 0; Finally, as the information between the Unpool layers propagates, the last layer restores the feature information of all characters.

7. The method for identifying group behavior based on dynamic and static features and multiple interactive networks according to claim 6, characterized in that: The number of fine-grained character information restoration modules is consistent with the number of coarse-grained key character extraction modules.

Images