Group behavior identification method based on dynamic and static characteristics and multiple interaction networks

Abstract

The invention discloses a group behavior recognition method based on dynamic and static characteristics and multiple interactive networks, and relates to the technical field of computer vision, comprising the following steps: preprocessing original video data and skeleton node data to obtain dynamic and static characteristics of participants; multi-level character features are obtained by extracting coarse-grained key character information and restoring fine-grained character information; the method comprises the following steps of: obtaining a complex space-time interaction feature by applying a Space-Transform and a Temporal-Transform, and obtaining the complex space-time interaction feature by applying the Space-Transform and the Temporal-Transform; predicting an identification result through data fusion; dynamic features are utilized to supplement static features, and group behaviors of participants are comprehensively and meticulously analyzed; in order to emphasize the key figure information, the key figure information is used as a consistency constraint to guide the focus of the space-time interaction network, and the recognition capability of the model on the space-time dynamic state of the key figure is enhanced.

Description

Technical Field

[0001] The present invention relates to the field of computer vision technology, and in particular to a group behavior recognition method based on dynamic and static features and multiple interactive networks. Background Art

[0002] Group behavior recognition has become a promising research direction in the field of computer vision due to its diverse applications. Its purpose is to identify the behavior of a group of people in a specific scene. It is widely used in application fields such as intelligent monitoring, content-based video retrieval, automatic video analysis and annotation, and motion analysis. It also has a positive role in promoting many industries such as transportation and criminal investigation.

[0003] There are three types of group behavior recognition methods: traditional methods and deep learning methods. Traditional group behavior recognition methods have many limitations and generally have average results. In recent years, due to the development of deep learning technology, research in this field has made significant progress. Deep learning-based group behavior recognition methods are divided into three categories: methods based on hierarchical spatiotemporal networks, methods based on deep learning relational models, and methods based on attention mechanisms.

[0004] 1) Methods based on hierarchical spatiotemporal networks

[0005] The method based on hierarchical spatiotemporal network first uses convolutional neural network to extract local character features, and then uses hierarchical recurrent neural network to gradually fuse feature sequences at different levels, and finally obtains the global features of the entire group for group behavior recognition. This type of method only extracts features at different levels in stages, and uses high-level features to judge group behavior, ignoring the interactive relationship between participants and between participants and the group.

[0006] 2) Methods based on deep learning relational models

[0007] Most methods based on deep learning relationship models use deep learning technology to model the interaction between participants and between participants and groups in multi-person scenarios. This type of method uses deep learning relationship networks to extract participant interaction information, making participant features more complete. However, different types of interaction features need to be extracted using targeted deep learning relationship models to ensure that more detailed and comprehensive participant interaction features are captured.

[0008] 3) Methods based on attention mechanism

[0009] The method based on the attention mechanism is the Vision Transformer (ViT) proposed by Google in 2020. This model does not require convolution and directly uses the Transformer to classify image block sequences. It has achieved results comparable to the current optimal convolutional network, but the computing resources required for training are greatly reduced compared to convolutional neural networks, which to a certain extent solves the challenges of high complexity and large computing workload in group behavior recognition tasks. Most of the existing solutions that use the attention mechanism to deal with unequal interactions in group activities cannot adjust the attention range according to downstream tasks, and lack flexibility and adaptability.

[0010] In view of the problem that in the prior art, static features of participants are not sufficient to fully characterize the behavior of participants, the existing method of using optical flow to extract dynamic features of participants is highly complex and unstable, and the model simply uses the attention mechanism to highlight key figures and cannot flexibly adjust the number of key figures, the present invention aims to provide a group behavior recognition method based on dynamic and static features and multiple interactive networks. Summary of the invention

[0011] In order to make up for the shortcomings of the existing technical problems, the present invention provides a group behavior recognition method based on dynamic and static features and multiple interactive networks, which uses dynamic features to supplement static features and comprehensively and meticulously analyze the group behavior of participants; by extracting coarse-grained key person information and restoring fine-grained person information, it is excluded that the interaction of non-key persons may cause the model to interfere with the correct action recognition; in order to emphasize the key person information, the key person information is used as a consistency constraint to guide the focus of the spatiotemporal interactive network, thereby enhancing the model's recognition ability of the spatiotemporal dynamics of key persons.

[0012] To achieve the above object, the present invention provides the following technical solutions:

[0013] A method for identifying group behavior based on dynamic and static features and multiple interactive networks, comprising the following steps:

[0014] (1) Preprocessing the original video data and skeleton node data;

[0015] (2) Inception-v3 is used to extract multi-scale features from the preprocessed video data, and then RoIAlign is used to extract the static features of all participants in each frame;

[0016] (3) The preprocessed skeleton node data is sequentially input into the spatial graph convolutional network and the temporal graph convolutional network to obtain the spatiotemporal motion features of the internal joints of the human body. The spatiotemporal motion features of the internal joints of the human body are then input into the TemporalAttention module to extract the overall dynamic features of the participant.

[0017] (4) After changing the dimension of the dynamic features of the participants as a whole so that their dimension is the same as the static feature dimension, the static feature is merged with the dynamic features of the participants as a whole after the dimension change to obtain the dynamic and static features;

[0018] (5) Use multiple coarse-grained key person extraction modules to extract information from multiple key persons, and then use multiple fine-grained person information restoration modules to restore information for all key persons. Add a GCN layer after the multiple fine-grained person information restoration modules to infer and calculate the interaction features of all participants after restoration;

[0019] (6) In the fine-grained reconstruction module and the coarse-grained key person extraction module, a Maxpool layer is added after each GCN layer that integrates the participant features, and the group features output by multiple maximum pooling layers are added and fused together to form the group-level features of the character interaction; the dynamic and static features in step (4) are also connected to a Maxpool layer to aggregate into group-level features; the group-level features of the character interaction are added and fused with the group-level features to obtain multi-level character features;

[0020] (7) After transposing the dimensions of the dynamic and static features, Sptial-Transformer is used to model the spatial context of all frames, and then Temporal-Transformer is used to model the temporal context of all frames to obtain complex spatiotemporal interaction features;

[0021] (8) Integrate multi-level character features with complex spatiotemporal interaction features to predict the probability of occurrence of each category of group behavior Using dynamic feature X D Predict the probability of each category occurring Add the probability results of the two predictions and select the highest result P.

[0022] In the present invention, the pretreatment in step (1) includes:

[0023] For a video input S, it is divided into K time blocks of equal time {S 1 ,S 2 ,...,S k}, the number of video frames in each time block is X t ,t=1,2,…,T, there are N bounding boxes for each frame Get image sequence Where 3 represents the number of channels, T represents T frames are extracted each time, H and W represent the height and width of the video frame respectively, and the input data is normalized;

[0024] The skeleton node data is obtained by HRNet and is defined as Among them, N represents that there are N participants in each frame, 3 represents that the skeleton node data has three dimensions, namely the X coordinate and Y coordinate of the node and the confidence of the skeleton node, N joint Each participant has N joint skeleton nodes, N is fixed in the experiment joint =18, the input data is normalized.

[0025] In the present invention, the process of extracting the dynamic features of the participants as a whole is as follows:

[0026] First, construct the spatial graph G of each participant at frame t t =(V t ,E t ), where V t = {v ti ∣v ti ,i=1,...,N joint} represents the participant joint node, and the edge set E t ={e ij |i,j=1,...,N joint , i≠j} represents the joint nodes vti and v tj The relationship between Represents the characteristics of each node, where N joint represents the number of nodes, C' represents the node attribute dimension. When constructing the adjacency matrix A, it is noted that the range of movement of human joints is limited, and only node v is considered. ti With its neighborhood set N(v ti )={v tj ∣d(v ti ,v tj )≤1,j≠i}; where d(v ti ,v tj ) represents the joint node v ti and v tj The distance between Input into the spatial graph convolutional network to obtain the spatial features X of the upper joint points inside the human body S And the updated adjacency matrix A S , the formula is as follows:

[0027]

[0028] Then, X S Put it into the time graph convolutional network to capture the spatiotemporal motion features X of the same joint point in different frames ST , create a time graph G based on the i-th node in different frames i =(V i ,E i), where V i = {v ti |t=1,2,...,T},E i ={e tt′ |t,t′=1,2,...,T,t′≠t}, similar to S-GCN which only considers the neighborhood of the joint point, only considers the three frames around the t-th frame, and the formula is as follows:

[0029]

[0030] Secondly, the spatiotemporal motion characteristics of the joints inside the human body are ST Put it into the Temporal Attention module to extract the dynamic features of the whole participant.

[0031] X ST ←Temporal Attention(X ST ) (4).

[0032] In the present invention, multiple coarse-grained key person extraction modules perform information extraction process on multiple key persons:

[0033] First, each participant node in the graph corresponds to a set of Where N is the number of participants, is the dynamic and static features of the ith participant, i.e., the dynamic and static features X in step (4) ID The i-th participant characteristics in ; The position feature of the ith participant is represented by the center coordinates; the dynamic and static features and the position features are fused into the participant features that are finally input into GCN; after the initial participant relationship is established, GCN is used to further reason about the relationship, and the original participant relationship X A The participant characteristics X are obtained through formula (7): B ;

[0034]

[0035] in, A is the adjacency matrix of the input graph, I is the identity matrix, yes , W is the learned weight matrix, and σ(·) is the activation function RELU;

[0036] Then, in order to pay more attention to the interactions between key participants and extract key participant information as the consistency constraint information of multiple interactions into the spatiotemporal interaction, the SAGpool method is used to extract coarse-grained key person information; the key to SAGpool is that it uses GCN to provide a self-attention score Score; Score represents the importance of different participants in each frame, and the calculation formula is as follows:

[0037]

[0038] Among them, Score∈R N×1 is the score obtained by GCN; select the top k nodes with the largest scores to form the final character selection mask Score mask ,in, The pooling ratio λ∈(0,1] is a hyperparameter that determines the number of nodes to be retained; idx=rank(Score,k), the function rank returns the index of the top K largest scores in Score, idx It is an index operation, Score mask =Score idx ;

[0039] Finally, according to the obtained Score mask , construct a coarser-grained relationship graph, and calculate the node feature X according to the character selection mask (l) And update the feature matrix X (l+1) and the adjacency matrix A (l+1) ,in, X idx,: is the row-wise indexed feature matrix, X (l+1) =X (l) ⊙Score mask , A idx,idx is the adjacency matrix indexed in row and column directions, and ⊙ denotes the broadcast element-wise product.

[0040] In the present invention, in the process of extracting information from multiple key figures, three coarse-grained key figure extraction modules are stacked, and the key figure features obtained by the last module are

[0041] In the present invention, the process of restoring information of all key characters using multiple fine-grained character information restoration modules is as follows:

[0042] First, the Unpool layer in the fine-grained character information restoration module puts the key nodes back to their original positions in the fine-grained relationship graph according to the indexes recorded by the key character extraction module, thereby restoring the fine-grained relationship graph; the GCN layer captures the interactions between the restored nodes;

[0043] Then, a jump connection is added between the SAGpool layer of the coarse-grained key character extraction module and the Unpool layer of the fine-grained character information restoration module to better reconstruct the fine-grained character relationship graph and transfer the topological structure and deep features of the relationship graph between the two modules. The specific formula is as follows:

[0044]

[0045] Among them, idx∈Z *k is the index of k key nodes, X (l) is the feature matrix of the first layer; X (l) The features X obtained by the three key character information extraction modules kv ; Represents the initial feature matrix when the Unpool layer in the fine-grained character information restoration module reconstructs the character relationship, and its value is 0; the function reconstruct converts X according to the corresponding index recorded in idx (l) The row vectors in are assigned to In, X (l+1) Represents the reconstructed fine-grained character relationship graph. The row vector of the record index in idx will be X (l) The row vectors in are updated, while the other row vectors remain 0;

[0046] Finally, as the information between the Unpool layers propagates, the last layer restores the feature information of all characters.

[0047] In the present invention, the number of fine-grained character information restoration modules is consistent with the number of coarse-grained key character extraction modules.

[0048] Compared with the prior art, the present invention has the following beneficial effects:

[0049] (1) In the present invention, the dynamic and static features of the participants are extracted respectively for the input of multimodal data (skeleton data and RGB data). The dynamic features can characterize the hidden features that are highly related to group activities, such as direction and speed. The method of extracting the dynamic features of the participants from the local to the whole using the motion trajectory module aims to provide supplementary information for the single static feature, improve the accuracy and stability of the model recognition, and combine the static features with the dynamic features to analyze the group behavior of the participants more comprehensively and meticulously.

[0050] (2) In the present invention, by extracting coarse-grained key person information and restoring fine-grained person information, the interaction of non-key persons is excluded, which may cause the model to interfere with correct action recognition; in the process of extracting coarse-grained key person information, the SAGpool method is used to extract coarse-grained key person information, which can flexibly adjust the number of key persons, increase the adaptability and maintainability of the overall model, and solve the problem that the traditional attention mechanism cannot adjust the focus scope in time according to task requirements; in the process of restoring fine-grained person information, the extracted key person information is reintegrated into the original fine-grained graph, maintaining a comprehensive understanding of the entire scene and utilizing the information of all participants, ensuring the comprehensiveness of the analyzed group behavior and enhancing the model's generalization ability for different scenarios.

[0051] (3) The present invention uses Spatial-Transformer and Temporal-Transformer to model temporal and spatial interactions respectively, and analyzes the temporal and spatial context of the behavior as supplementary information for the above-mentioned character interaction characteristics, so as to have a deeper understanding of the dynamic changes and evolution process of group behavior; in order to emphasize the key character information, the key character information is used as a consistency constraint to guide the focus of the spatiotemporal interaction network, thereby enhancing the model's ability to recognize the spatiotemporal dynamics of key characters. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] Figure 1 This is a framework diagram of the method of the present invention.

[0053] Figure 2 It is a schematic diagram of the motion trajectory module in the present invention.

[0054] Figure 3 This is a schematic diagram of coarse-grained key person extraction in the present invention.

[0055] Figure 4 It is a schematic diagram of multi-level character feature extraction in the present invention.

[0056] Figure 5 Schematic diagram of the Spatial Transformer and Temporal Transformer structure.

[0057] Figure 6 This is an example diagram of the volleyball data set in the present invention.

[0058] Figure 7 This is an example diagram of the collective activity data set in the present invention.

[0059] Figure 8 It is a schematic diagram of the influence of the motion trajectory module on the accuracy in the present invention.

[0060] Fig. 9 It is the confusion matrix of character interaction and spatiotemporal interaction in the present invention.

[0061] Fig.10 t-SNE classification visualization diagram of different parts of the present invention.

[0062] Fig.11 This is a comparison diagram of the confusion matrix of CAD in the present invention. DETAILED DESCRIPTION

[0063] The technical solutions in the embodiments of the present invention will be described clearly and completely below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than all the embodiments.

[0064] In order to identify complex group behaviors in different scenarios, a group behavior recognition method based on dynamic and static features and multiple interaction networks is proposed. Figure 1-11 , the framework of dynamic and static features and multiple interaction networks, DSMI, will be introduced in detail. First, the preprocessing of the original input data and the extraction of dynamic and static features are introduced; then, the character interaction network and spatiotemporal interaction network are described, and the selection rules of key figures and the formation of multiple interaction consistency constraints are explained; finally, the multiple interaction features are fused, and the fused features are scored and fused with the dynamic features for identifying group behavior.

[0065] Figure 1 The method framework diagram is shown. Figure 1 ×3 in the middle means that the key character extraction module and the following Maxpool layer are stacked three times, and the detail restoration module and the following Maxpool layer are stacked three times. For details, see Figure 4 . The framework consists of three parts. The first part is feature extraction. The RGB modal data and human skeleton node modal data are respectively input into the convolutional neural network and the motion trajectory module to obtain the dynamic and static features of the participants. The second part introduces the consistency constraints of adaptive multiple interactions. The dynamic and static features of the first part are input into the multiple interaction network to extract the character and spatiotemporal interaction features respectively, and the key character information in the character interaction is input into the spatiotemporal interaction network as a consistency constraint, so that the overall network can adaptively adjust the scope of key characters and enhance the network accuracy and flexibility. The third part introduces the process of judging group behavior at the level of feature fusion and classification results. The classification results of group behavior obtained by fusing the character interaction features with the spatiotemporal interaction features are later fused with the classification results obtained by the dynamic features to identify group behavior.

[0066] 1. Data preprocessing and dynamic and static feature extraction

[0067] The processing of the raw input data and the extraction of dynamic and static features of the participants using a convolutional neural network and a motion trajectory module will be described.

[0068] 1.1 Data Preprocessing

[0069] In order to better put the original data into the deep network to extract features, the original video input and skeleton node input need to be preprocessed first.

[0070] For a video input S, it is divided into K time blocks of equal time {S 1 ,S 2 ,…,S k}, the default time interval is 10 frames. The number of video frames in each time block is X t ,t=1,2,…,T, there are N bounding boxes for each frame Get image sequence The data is input into the DSMI model, where 3 represents the number of channels, T represents extracting T frames each time, H and W represent the height and width of the video frame respectively, and the input data is normalized.

[0071] The skeleton node information is obtained by HRNet and is defined as Among them, N represents that there are N participants in each frame, 3 represents that the skeleton node data has three dimensions, namely the X coordinate and Y coordinate of the node and the confidence of the skeleton node, N joint Each participant has N joint skeleton nodes, N is fixed in the experiment joint =18, the input data is normalized.

[0072] 1.2 Extraction of dynamic and static features

[0073] The process of group behavior recognition requires analyzing the actions of each participant, so it is necessary to extract the characteristics of the participants. The characteristics of participants are divided into static and dynamic. Static characteristics are mainly appearance characteristics, while dynamic characteristics are mainly characteristics generated by the movement of participants. For example, in a volleyball game, the "winning point" usually occurs when a specific player spikes the ball and the other player fails to catch the ball. The analysis of a player's spike and the other player's failure to catch the ball is based on the appearance characteristics of the players. In the time period before and after the key judgment actions of spiking and not catching the ball, other actions will occur. It is necessary to continuously analyze the actions over a period of time to finally judge the group behavior. Therefore, the dynamic and static characteristics of the participants are very important for judging group behavior.

[0074] In view of the fact that previous methods using optical flow or three-dimensional networks to extract dynamic features of participants are prone to interference and highly complex, the participant skeleton nodes are used to extract dynamic features. The input of the skeleton nodes is easy to obtain, simple and lightweight. After inputting them into graph convolution, accurate and stable participant motion features can be obtained, thereby improving the model's prediction ability.

[0075] (a) Extraction of static features

[0076] First, Inception-v3 extracts more comprehensive multi-scale features. Then RoIAlign extracts the static features of all participants in each frame and obtains X I ∈R T×N×C×K×K , where C represents the number of channels, N represents the number of participants in each frame, and K×K is the size of the feature map of each participant.

[0077]

[0078] (b) Extraction of dynamic features

[0079] like Figure 2 As shown in the figure, the motion trajectory module is composed of S-GCN, T-GCN and TemporalAttention. S-GCN and T-GCN extract the local motion information of the internal joints of the participants, and TemporalAttention extracts the overall motion information of the participants.

[0080] First, construct the spatial graph G of each participant at frame t t =(V t ,E t ), where V t = {v ti ∣v ti ,i=1,…,N joint} represents the participant joint node, and the edge set E t ={e ij |i,j=1,...,N joint , i≠j} represents the joint nodes vti and v tj The relationship between Represents the characteristics of each node, where N joint represents the number of nodes, C' represents the node attribute dimension. When constructing the adjacency matrix A, it is noted that the range of movement of human joints is limited, and only node v is considered. ti With its neighborhood set N(v ti )={v tj ∣d(v ti ,v tj )≤1,j≠i}. Among them, d(v ti ,v tj ) represents the joint node v ti and v tj The distance between Input into the spatial graph convolutional network (S-GCN) to obtain the spatial features X of the upper joints inside the human body S And the updated adjacency matrix A S , the formula is as follows:

[0081]

[0082] Then, X S Put it into the temporal graph convolutional network (T-GCN) to capture the spatiotemporal motion features X of the same joint point in different frames ST , which is different from S-GCN in that the time graph G is created based on the i-th node in different frames i =(V i ,E i ), where V i = {v ti |t=1,2,…,T},E i ={e tt′ |t,t′=1,2,…,T,t′≠t}, similar to S-GCN which only considers the neighborhood of the joint point, only considers the three frames around the t-th frame, and the formula is as follows:

[0083]

[0084] Secondly, the spatiotemporal motion characteristics of the joints inside the human body are ST The Temporal Attention module is used to extract the motion information of the participants as a whole. The Temporal Attention module can dynamically assign different attention weights to the input of different frames, so that the model can focus more on important frames, thereby accurately and effectively extracting the overall motion information of the participants.

[0085] X ST ←Temporal Attention(X ST ) (4)

[0086] Finally, in order to perform feature fusion with the static features of the participants extracted by Inception-v3, X ST Perform dimension changes as shown below:

[0087]

[0088] The static feature X I and dynamic feature X D After adding the corresponding elements, we get the total characteristics of the participants. The formula is as follows:

[0089]

[0090] In general, S-GCN and T-GCN respectively model the spatial features of different skeleton points in the same frame and the temporal features of the same skeleton point in different frames, which belong to the local motion information of the skeleton inside the participant, while the overall motion information of the participant is extracted through the Temporal Attention module.

[0091] The first stage aims to extract the characteristics of the participants, including the static appearance characteristics and dynamic motion characteristics of the participants. Such feature extraction can analyze the group behavior of the participants more comprehensively and meticulously, and provide more complete participant information for modeling multiple interactions between participants in the second stage.

[0092] 2. Multiple interaction networks based on consistency constraints

[0093] In group behavior recognition, the spatiotemporal background of some participants is also very complex and diverse. For example, in volleyball games, in R-set and L-pass actions, participants collaborate more, and the spatiotemporal context information does not change much, that is, the amount of character interaction information is large, and the amount of spatiotemporal interaction information is small; in R-spike and R-winpoint actions, the interaction between participants is not significant within a certain period of time, but the spatiotemporal context information has a significant auxiliary effect on the recognition of participants' actions. Therefore, independent and clear modeling of characters and spatiotemporal interactions enables the model to capture the behavior of participants more comprehensively and improve the recognition accuracy of the model. The interaction of non-key characters may interfere with the model's recognition of correct actions. The model needs to highlight the interaction of key characters and suppress the interaction of non-key characters. The key character information extracted from the character interaction is input into the spatiotemporal interaction as a consistency constraint, so that the overall model can adaptively adjust the scope of key characters according to the task.

[0094] In order to solve the problems of incomplete interaction features and unequal interaction relationships between participants, a multiple interaction network based on consistency constraints is proposed, which independently models participant interactions and spatiotemporal interactions, and adds consistency constraints between the two interactions, so that the overall model can adaptively adjust the scope of key figures and more effectively adapt to and optimize the execution of downstream tasks. This will be explained below.

[0095] like Figure 3 As shown, Figure 3 (a) Schematic diagram of coarse-grained key figure extraction. Figure 3(b) is a schematic diagram of fine-grained character restoration. The coarse-grained graph is restored to its original position. When restoring the character information in the reverse process, adding Skipconnection can transfer the topological structure and deep features of the relationship graph from the coarse-grained key character extraction module to the fine-grained character information restoration module. The Key-actor Graph* in the figure represents the key character graph obtained after the yellow Key-actor Graph is inferred by GCN.

[0096] 2.1. Character Interaction

[0097] (a) Coarse-grained key person extraction

[0098] In the human visual perception system, certain areas in the image have a decisive influence on the decision-making process. Take a volleyball game as an example. When a player performs a spike, the line of sight tends to focus on the player who performs the key action, rather than other defensive players on the court. These players who perform the main actions are called key figures, and their interactions are crucial to understanding the behavior of the entire group. The interactions between non-key figures are less important and may even interfere with the final prediction results. Therefore, it is necessary to highlight the interactions of key figures and suppress the interactions of non-key figures during the analysis process.

[0099] To address the problem that the traditional attention mechanism cannot adjust the scope of attention in time according to task requirements, the SAGpool method is used to extract coarse-grained key person information, which can flexibly adjust the number of key persons and increase the adaptability and maintainability of the overall model.

[0100] First, when the distance between participants is closer, their connection and mutual influence are more significant, which helps to make relational reasoning more effective. Therefore, each participant node in the graph corresponds to a set of Where N is the number of participants, is the dynamic and static characteristics of the ith participant, that is, X ID The i-th participant feature in . The location feature of the ith participant is represented by the center coordinates. The dynamic and static features and the location features are fused into the participant features that are finally input into GCN. After the initial participant relationship is established, GCN is used to further reason about it. The original participant relationship X A The participant characteristics X are obtained through formula (7): B .

[0101]

[0102] in, A is the adjacency matrix of the input graph, I is the identity matrix, yes , W is the learned weight matrix, and σ(·) is the activation function RELU.

[0103] Then, in order to pay more attention to the interaction between key participants and extract key participant information as the consistency constraint information of multiple interactions into the spatiotemporal interaction, the SAGpool method is used to extract coarse-grained key person information. The key to SAGpool is that it uses GCN to provide a self-attention score Score. Score represents the importance of different participants in each frame, and the calculation formula is as follows:

[0104]

[0105] Among them, Score∈R N×1 is the score obtained by GCN. Select the top k nodes with the largest scores to form the final character selection mask Score mask ,in, The pooling ratio λ∈(0,1] is a hyperparameter that determines the number of nodes to be retained. idx=rank(Score,k), the function rank returns the index of the top K largest scores in Score, idx It is an index operation, Score mask =Score idx .

[0106] Finally, according to the obtained Score mask , construct a coarser-grained relationship graph, and calculate the node feature X according to the character selection mask (l) And update the feature matrix X (l+1) and the adjacency matrix A (l+1) ,in, X idx,: is the row-wise indexed feature matrix, X (l+1) =X (l) ⊙Score mask , A idx,idx is the adjacency matrix indexed in row and column directions, and ⊙ denotes the broadcast element-wise product.

[0107] As information propagates between SAGpool layers, the last layer obtains the most critical character feature information. A total of three key character information extraction modules are stacked, and the key character features obtained by the last module are The consistency constraints of external key figures are passed into the spatiotemporal interaction network. The process is shown as follows Figure 3 (a) shown.

[0108] (b) Fine-grained character information restoration

[0109] In order to maintain a comprehensive understanding of the entire scenario and utilize the information of all participants, it is necessary to reintegrate the information of key figures into the original fine-grained graph. This can ensure the comprehensiveness of the analyzed group behavior and enhance the model's generalization ability to different scenarios.

[0110] Fine-grained character information restoration is the inverse operation of coarse-grained key character extraction, aiming to reconstruct a fine-grained relationship graph. Figure 3 As shown in (b), first, the Unpool layer in the fine-grained reconstruction module puts the key nodes back to their original positions in the fine-grained relationship graph according to the indexes recorded by the key person extraction module, thereby restoring the fine-grained relationship graph. The GCN layer captures the interactions between the restored nodes.

[0111] Then, adding a jump connection between the SAGpool layer of the coarse-grained key character extraction module and the Unpool layer of the fine-grained character information restoration module can better reconstruct the fine-grained character relationship graph. It can transfer the topological structure and deep features of the relationship graph between the two modules. The specific formula is as follows:

[0112]

[0113] Where idx∈Z *k is the index of k key nodes, X (l) is the feature matrix of the lth layer. (l) The features X obtained by the three key character information extraction modules kv . Represents the initial feature matrix when the Unpool layer in the fine-grained character information restoration module reconstructs character relationships, and its value is 0. The function reconstruct converts X to (l) The row vectors in are assigned to In, X (l+1) Represents the reconstructed fine-grained character relationship graph. The row vector of the record index in idx will be X (l) The row vectors in are updated, while the other row vectors remain 0.

[0114] Finally, as the information between the Unpool layers propagates, the last layer restores the feature information of all characters. It should be noted that the features X of all characters derived and restored by GCN are C With the original X B Although the shape is the same, X C It will highlight the characteristics of key figures and represent group behavior more accurately. The number of blocks in the fine-grained character restoration module and the key figure information extraction module is consistent, which is 3.

[0115] A GCN layer is added after the three fine-grained character information restoration modules to infer and calculate the interaction features of the N participants after restoration. C After the last GCN, we get X′ D .

[0116] (c) Extraction of multi-level character features

[0117] Group behavior is composed of multiple participant behaviors, and participant behaviors are guided by group behavior. If we only focus on participant-level features, the model is easily limited to specific individuals and difficult to generalize to new groups and scenarios. In order to facilitate multi-level analysis and meet the needs of different tasks, we extract features at two levels at the same time. The coarse-grained key person extraction and fine-grained person information restoration obtain the interaction information at the participant level. The group-level features can be obtained by aggregating the participant-level features through the maximum pooling layer.

[0118] First, in the fine-grained reconstruction module and the coarse-grained key person extraction module, a Maxpool layer is added after each GCN layer that integrates the participant features. This is to effectively aggregate the participant-level features and generate more comprehensive group-level features.

[0119] Then, the group features output by multiple maximum pooling layers are added together to form the features at the character interaction group level. The dynamic and static features X of a frame of participants extracted in the first stage ID A Maxpool layer is also connected to aggregate the group-level features. With X ID Add and blend together.

[0120]

[0121] Finally, the above interactions between participants are all performed on the same frame. Therefore, the characteristics of the character interaction of participants on frame T can be expressed as The process diagram is as follows Figure 4 .

[0122] The character interaction module extracts coarse-grained key character information through the SAGpool method, and uses the inverse operation to restore it to the fine-grained participant graph to fully capture the interaction characteristics in group behavior. At the same time, the key character information is used as a consistency constraint to guide the focus of the spatiotemporal interaction network, enhancing the model's ability to recognize the spatiotemporal dynamics of key characters. The present invention extracts multi-level interaction features from the participant level and the group level, which not only improves the generalization ability of the model, but also ensures the performance stability in new situations, providing a more in-depth and comprehensive perspective for group behavior recognition.

[0123] 2.2. Space-time interaction

[0124] The spatiotemporal context of participants in group behavior has an important impact on their behavior judgment, and spatiotemporal context information is crucial for a comprehensive understanding of group behavior. Spatial-Transformer and Temporal-Transformer are used to model temporal and spatial interactions respectively, and the temporal and spatial context of the behavior is analyzed as supplementary information to the above-mentioned character interaction features, so as to gain a deeper understanding of the dynamic changes and evolution of group behavior.

[0125] (a) Spatial interaction

[0126] First, the participant characteristics output from the first stage After dimensional swapping, we get:

[0127]

[0128] Then, when modeling spatial interactions, the time dimension T is regarded as a batch dimension, the spatial interactions of N characters in the same frame are considered, and the spatial-Transformer is applied to model the spatial context of all frames. ID The input map is the participant query Q (t) , the key character features X extracted by the character interaction module kv Mapped to key K (t) Sum value V (t) , as the consistency constraint of multiple interactions, is passed into the Sptial-Transformer, which allows the network to pay more attention to the spatial interaction information of key figures. The mapping is done through the weight matrix Completed, they are all shaped like D e ×D e The similarity between the participant features and all key person features is calculated to generate a similarity matrix.

[0129] Next, in order to make the gradient update more stable during training, each element of the similarity matrix is ​​divided by Q (t) and K (t) The dimension size is , and the result is normalized using the Softmax function.

[0130] Finally, the normalized similarity matrix is ​​compared with V (t) After doing a MatMul operation with V (t) Make a residual connection to get the attention output V' (t) The residual connection can retain the key information of the original input and improve the expressiveness of the model. The calculation formula is as follows:

[0131]

[0132] The output of attention V' (t) Input the feedforward neural network (FFN) to obtain the spatial feature V" of the tth frame (t) , the feature maps of all frames {V" (t) |t=1,…,T} are stacked together and the shape becomes The final spatial interaction feature is The structural diagram is as follows Figure 5 (a).

[0133] (b) Time Interaction

[0134] After using Spatial-Transformer to model the spatial interaction of group behavior, the same operation method is used to model the temporal interaction. The difference between Temporal-Transformer and Spatial-Transformer is that it regards the spatial dimension as a batch dimension, that is, it considers the temporal interaction of the same person in different frames. Temporal-Transformer is used to model the temporal context of all frames. The schematic diagram of Temporal-Transformer structure is shown in the figure below. Figure 5 (b).

[0135] First, use represents the output time dynamic features of the nth individual, and the generated feature map {V" (n) |n=1,2,…,N} stacked together, the shape becomes

[0136] Then, in order to interact with the character for feature fusion, V t To perform dimension conversion:

[0137]

[0138] Finally, the spatiotemporal interaction features of N participants are aggregated into group-level features through the maximum pooling layer, and the formula is:

[0139]

[0140] The above multi-interaction modeling stage is the second stage of the overall model, which can obtain multi-level character interaction features. and complex spatiotemporal interaction characteristics Adding consistency constraints to character interactions and spatiotemporal interactions allows the second stage to focus on the interaction information of key characters. This method independently and clearly models character and spatiotemporal interactions, allowing the model to learn more comprehensive interaction details to better adapt to complex people and different scenarios.

[0141] 3. Comprehensive fusion process

[0142] First, the character interaction Interacting with space and time For feature fusion, the formula is as follows:

[0143]

[0144] Then, use the fused feature X interaction Predict the probability of each category of group behavior occurring At the same time, the dynamic feature X output by the motion trajectory branch D Also predict the probability of each category occurring Add the probability results of the two predictions and select the highest result P.

[0145]

[0146] Among them, N c The total number of categories representing group behavior.

[0147] Finally, the category corresponding to P is used as the prediction result of the final group behavior.

[0148] 4. Method evaluation

[0149] In order to evaluate the proposed method, ablation experiments and comparative experiments were conducted on the public datasets Volleyball Dataset (VD) and Collective Activity Dataset (CAD) to verify the effectiveness of the group behavior recognition method based on dynamic and static features and multiple interactive reasoning. First, the implementation details of the dataset and the method are briefly introduced. Next, the method of the present invention is compared with the previous state-of-the-art methods. Then, in order to demonstrate the respective contributions of different parts, some ablation studies are conducted. Visualization results and corresponding analysis are given.

[0150] 4.1) Metrics

[0151] Two commonly used accuracy measures, Multi-Class Accuracy (MCA) and Mean Per-Class Accuracy (MPCA), are used to evaluate the performance of classification models. MCA is suitable for overall model performance evaluation, while MPCA is more suitable for detailed analysis of the classification of each category, especially for datasets with imbalanced categories. Based on the above description, MCA is used to evaluate the performance of the Volleyball dataset, and MPCA is used to evaluate the performance of the Collective dataset. The accuracy calculated using MCA or MPCA is called precision.

[0152] MCA refers to the average classification accuracy of all categories in a multi-classification task. It is a simple and intuitive performance indicator, calculated as follows:

[0153]

[0154] Where N is the number of categories, TP i Indicates the number of true positive examples of the i-th category, FP i Represents the number of false positive examples in the i-th category.

[0155] MPCA refers to the average classification accuracy of each category in a multi-classification task. Compared with MCA, MPCA pays more attention to the classification performance of each category, especially for cases with imbalanced categories. The calculation method is as follows:

[0156]

[0157] Among them, Acc i Represents the classification accuracy of the i-th category.

[0158] 4.2) Dataset Introduction

[0159] The DSMI framework is evaluated on the following two public GAR datasets that provide tracking annotations and bounding boxes.

[0160] Volleyball Dataset (VD): The dataset consists of 4830 video clips of 55 volleyball games (3493 clips are used for training and 1337 clips are used for testing), each clip has 41 frames. The middle frame of each clip contains the bounding box coordinates, a single action label, and a group activity label. The single action label contains 9 actions: setting, digging, falling, jumping, blocking, moving, spiking, waiting and standing. The group activity label contains 8 activities, namely right setting (right_setting), right passing (right_passing), right spiking (right_spiking), right win point (right_winpoint), left setting (left_setting), left passing (left_passing), left spiking (left_spiking) and left win point (left_winpoint). Figure 6 As shown, for fair comparison with the GAR method, the coordinates and training / test splits of each participant are used as provided in [1, 2, 4, 6].

[0161] Collective Activity Dataset (CAD): This dataset consists of 44 video clips shot by a low-resolution handheld camera with dynamic perspectives, with frame numbers ranging from 194 to 1814. The dataset contains five different collective activity labels, namely crossing, waiting, queuing, walking, and talking, six different individual action labels, namely NA, crossing, waiting, queuing, walking, and talking, and eight different individual posture labels (not used in the work). The middle frame of every ten frames contains bounding box coordinate annotations and individual action labels. The actions of most individuals in the same scene determine the group activity. According to the equivalent experimental setting in [1], 2 / 3 of the dataset is selected for training and the rest is used for testing. Figure 7 As shown in the figure, following the method of [5,6,8], the crossing and walking classes are merged into the moving class.

[0162] 4.3) Experimental details

[0163] The Inception-v3 model pre-trained on ImageNet was selected as the feature extraction network, following the strategy widely used in GAR. For VD and CAD, T = 3 frames of input models were sampled when training the models, and T = 9 frames were sampled when testing the models. When extracting static features, the crop size of RoIAlign was 5×5, and the dimension of the participant features was 1024. When extracting dynamic features, the number of input channels and output channels of S-GCN and T-GCN were set to 3 and 256, respectively, and the number of attention heads in the temporal attention module was set to 8. For fair comparison with other papers, the maximum number of participants N was set to 12 in VD and 13 in CAD. For VD, the initial learning rate (lr) was set to 2e-4. lr decreased with the increase of epochs, and dropped to 1e-4, 5e-5, and 1e-5 at the 40th, 80th, and 120th epochs, respectively. For CAD, lr started from 1e-4 and remained unchanged. On both datasets, the optimizer of the model is ADAM, the loss function is the cross entropy function, the MCA evaluation standard is used for VD, and the MPCA evaluation standard is used for CAD. ARG is selected as the Baseline method. The experimental environment and configuration are shown in Table 1.

[0164] Table 1 Experimental environment

[0165]

[0166] 4.4) Ablation Experiment

[0167] In order to demonstrate the respective contributions of different parts to the method, some ablation studies were conducted, using MCA and MPCA as experimental evaluation indicators on VD, and MPCA as experimental evaluation indicators on CAD. The specific methods are as follows:

[0168] Comparison of composition and accuracy of different baselines:

[0169] B1-Baseline Model: This base model includes Inception-v3 network, RoIAlign, Embedding layer and Softmax classification layer.

[0170] B2-Baseline Model+Motion: This baseline consists of the basic model and motion trajectory modules.

[0171] B3-Baseline Model+Motion+Actor Interaction: This baseline consists of the basic model, motion trajectory module and character interaction module.

[0172] B4-Baseline Model+Motion+ST Interaction: This baseline consists of the basic model, motion trajectory module and spatiotemporal interaction module.

[0173] Ours method: The first stage of the complete model consists of the Inception-v3 network and the motion trajectory module. The second stage includes the character interaction and spatiotemporal interaction modules. The key character information extracted by the character interaction module is passed to the spatiotemporal interaction module as an external constraint. Finally, the classification results of the dynamic features extracted by the motion trajectory module are fused with the classification results of multiple interaction features to judge group behavior.

[0174] Table 2 Comparison of the accuracy of ablation experiments in each part

[0175]

[0176] Note: "Motion" in the table represents the motion trajectory module

[0177] All training settings are the same except for the model structure, and the table shows that the full model outperforms these baselines. These comparisons are analyzed below:

[0178] As shown in Table 2. The comparison between baselines B2 and B1 confirms the effectiveness of the motion trajectory module in the feature extraction stage. The MCA of baseline B2 on VD and CAD is improved by 4.0% and 2.5% respectively compared with baseline B1, proving that Motion TrajectoryModule can supplement dynamic motion information for GAR. The comparison between baselines B3 and B2 confirms the effectiveness of the character interaction module. The MCA of baseline B3 on VD is improved by 0.7% compared with baseline B2, and the MPCA on CAD is improved by 1.2%. It is proved that character interaction information helps the model identify unequal interaction relationships between participants and distinguish similar behaviors. The comparison between baselines B4 and B2 confirms the effectiveness of the spatiotemporal interaction module. The MCA of baseline B4 on VD is improved by 0.6% compared with baseline B2, and the MPCA on CAD is improved by 0.4%. It is proved that the contextual information of participants extracted by this module helps the model distinguish group behaviors in complex environments. The comparison of the overall model with baselines B3 and B4 confirms the effectiveness of multiple interactions. The recognition accuracy of the overall model is higher, which shows that comprehensive and complete interactions can enhance the adaptability of the model.

[0179] Effect analysis of motion trajectory module: The accuracy of motion trajectory module is compared on VD and CAD to prove the effectiveness of other modules, such as Figure 8 As shown by Figure 8 (a) Figure 8(b) It can be seen that the motion trajectory module can help the model extract the hidden motion information of the participants as supplementary information of the static features, and a high recognition accuracy can be obtained when only the dynamic and static features of the participants are used to judge group behavior. Figure 8 (c) Figure 8 (d) It can be seen that, on the one hand, the input of dynamic features into the multiple interaction network is conducive to the model extracting more interaction details with the help of the participants' motion information; on the other hand, the fusion of the dynamic feature classification results and the classification results of multiple interaction features significantly improves the accuracy of group behavior recognition (accuracy is shown in Table 2), which shows that dynamic features can capture effective key motion details, thereby optimizing the classification results.

[0180] Multiple interaction validity analysis: The confusion matrix of baseline B3 and baseline B4 on VD proves that the person interaction and spatiotemporal interaction network can effectively capture the interaction features and improve the model recognition accuracy. Fig. 9 As shown, Fig. 9 (a) is the confusion matrix of character interaction. Fig. 9 (b) is the confusion matrix of spatiotemporal interaction. "L" and "R" are the abbreviations of "left" and "right" in group behavior labels. In the two types of behaviors, R_winpoint and L-set in the VD dataset, the accuracy of human interaction and spatiotemporal interaction is equal, and the judgments on other categories form complementary advantages. Human interaction can capture the unequal interactions between people, and spatiotemporal interaction focuses on capturing the interaction between participants and complex spatiotemporal. Multiple interactions make the interactions captured by the model more detailed and complete. Even in an environment with complex spatiotemporal backgrounds or cluttered crowds, it can still accurately identify group behaviors, improving the adaptability and stability of the model.

[0181] Visualization of the classification of different parts: Fig.10 It clearly demonstrates the classification ability of the model after adding characters and spatiotemporal interactions on VD. Fig.10 (a) shows the classification of baseline B2. It can be seen that the classification boundaries of R_spike, R_pass and R_set are not clear, and some L_pass classes are misclassified as L_spike classes. Fig.10 (b) shows the classification of baseline B3. It can be seen that the classification boundaries between R_set and R_pass are not clear. Compared with baseline B2, R_spike classification is clearer. It can be seen that after adding character interaction information, the model can learn more differences between participants' behaviors. Fig.10 (c) shows the classification of baseline B4, where we can see the classification degree and Fig.10 (b) Quite similar. Fig.10(d) shows the classification of the method. It can be observed that the classification boundaries of the L_set and R_winpoint classes are clearer than all previous baselines. From the perspective of the action itself, the R_set and R_pass behaviors are both completed by a key participant, and the action similarity is high. Therefore, the classification boundaries of the R_set and R_pass classes are not very clear under different baselines. In this case, the overall model makes the inter-class spacing of different categories larger, while the intra-class spacing of the same category becomes smaller, which shows that the overall model is more stable and has stronger classification ability than other baseline models.

[0182] Analysis of consistency constraints: The present invention conducts ablation experiments on VD and CAD to verify the effectiveness of key character information extracted from character interaction as consistency constraints. The results are shown in Table 3. When the query (Q), key (K), and value (V) are homologous, that is, they all come from the dynamic and static features of the participants in the first stage, the performance of the model is lower than the method of passing key character information as keys and values ​​into the Spatial-transformer. Specifically, on VD, using Q = K = V = X ID The MCA and MPCA of the method were 93.1% and 93.4% respectively, while the accuracy of the method of passing the key person information as key and value into the Spatial-transformer was increased to 93.8% and 94.1% respectively. This shows that the key person information has a significant advantage in capturing the unequal interaction relationship and complex spatiotemporal dynamics between characters, and keeping the key person who focuses on the same multiple interactions will improve the stability of the model. ID ,V=X kv When the method constrains the consistency of multiple interactions, the accuracy is improved compared to the method without consistency constraints, but the performance is not as good as the second experimental setting. This is because Q captures the dynamic and static characteristics of the participants, while K and V are derived from the characteristic information of key figures. This design enables the model to more effectively capture the rich spatiotemporal contextual features of key figures. When K and V share the same information source, the model can more accurately receive the effective constraint information of key figures, thereby achieving better performance.

[0183] Table 3 Comparison of key person information transmission methods

[0184]

[0185] Key Person Extraction Analysis: The present invention performs an ablation experiment on the number of modules for extracting key persons in a character interaction network on VD. As shown in Table 4, the accuracy of the three modules used is the highest compared with the use of 1, 2, and 4 modules for extracting key persons. MCA is 0.7% higher than the use of 2 modules and 0.6% higher than the use of four. The reason is that if there are too few modules, the most critical person will not be screened out, and the influence of non-critical persons is still very large, which interferes with the identification of group behavior; and if there are too many modules, it will cause over-screening of key persons, so that some key persons are not noticed, which is not conducive to the model reasoning and identification of correct behavior. The module for extracting key persons is composed of a self-attention pooling layer and a graph convolution layer. By stacking different numbers of self-attention pooling layers in the character interaction network, the attention range is dynamically adjusted, and the most suitable number of key persons is flexibly screened out according to the needs of specific tasks. This can enhance the flexibility and adaptability of the model and ensure that the optimal key person identification effect can be achieved in complex tasks.

[0186] Table 4 Comparison of the number of blocks in the coarse-grained key figure module

[0187]

[0188] Analysis of comprehensive fusion methods: The present invention conducted an ablation experiment on the fusion methods at the feature and result levels on VD, aiming to verify the effectiveness of fusing multiple interactions at the feature level and fusing multiple interaction classification results with dynamic feature classification results at the result level. The experimental results are shown in Table 5. The feature fusion adopted three typical methods: Addition, Concat, and Gating. The MCA that uses the Addition method to fuse multiple interaction features is 1.3% and 0.8% higher than the Concat method and the Gating method. In addition, the fusion at the result level also enhances the classification performance of the model. This is because the comprehensive fusion strategy can make full use of the complementarity of different features, and allows the model to be adjusted according to the importance of the features, increasing the flexibility of the model.

[0189] Table 5 Comparison of feature fusion methods

[0190]

[0191] 4.5) Comparison with previous methods

[0192] In this section, the proposed DSMI method is compared with previous methods on the above two public GAR datasets.

[0193] (a) Results on the volleyball dataset: The experimental results are shown in Table 6. The “MCA-VD” and “MPCA-VD” columns in the table compare the proposed method with the latest VD methods. It can be observed that compared with the worst performing SBGAR[9] method, the proposed method improves MCA by 26.9% and MPCA by 26.5%. StagNet[3] understands group activities in videos by combining spatiotemporal attention mechanism and semantic graph modeling. Compared with the previous methods that only used RNN and LSTM, StagNet achieved significant improvements. DIN[7] consists of a dynamic relationship (DR) module and a dynamic walk (DW) module. Compared with RNN-based models[9-10], DIN not only consumes less computational overhead but also performs better, mainly because it can better represent the joint modeling of spatiotemporal interactions; DIN is also better than GNN-based methods[2,4,11], mainly because of the dynamics of DR and DW modules. The proposed method (MCA-93.8%) is significantly improved compared with the methods that also use Inception-v3. Compared with the method of using additional optical flow input [5], human skeleton node data is used to capture the movement of the participant. This approach does not require additional calculation of optical flow and has lower computational complexity. At the same time, the backbone network used is a two-dimensional network, and its computational complexity is much lower than that of using an I3D network.

[0194] Table 6 Comparison of the accuracy of the proposed method and some classic GAR methods on VD

[0195]

[0196] Note: Bold represents the method with the highest accuracy in the table, and “-” represents that the MPCA of this method is not public.

[0197] (b) Results on the collective activity dataset: The experimental results are shown in Table 7. Specifically, on the collective activity dataset, the proposed method achieved a SOTA accuracy of 97.4%. Compared with non-deep methods, the proposed method is much more accurate; when compared with previous studies TCE+STBiP and Dual-AI that use Inception-v3 as the backbone network and adopt the same dataset configuration (i.e., merging the categories crossing and walking into moving), the proposed method achieved significant improvements in accuracy of 2.3% and 0.9%, respectively. In order to further demonstrate the effectiveness of the proposed method, the classification results were compared with DIN Fig.11 (a) and MLST Fig.11 (b) Compare the results of the methods, such as Fig.11 As shown, although the method of the present invention Fig.11(c) There are cases where the moving class is mistakenly identified as the waiting class, which may be due to the similarity of the actions of the two classes when crossing the road. The proposed method shows a significant performance improvement in identifying the waiting class, which is mainly due to the rich contextual information provided by spatiotemporal interactions in multiple interactions. In addition, the proposed method achieves 100% accuracy in the queuing and talking classes, further demonstrating the effectiveness of the DSMI framework.

[0198] Table 7 Comparison of the accuracy of the method of the present invention and some classic GAR methods on CAD

[0199]

[0200] Note: "*" represents the method of merging the "Crossing" class and the "Walking" class into the "Moving" class in CAD.

[0201] What is described above is only a preferred specific implementation manner of the present invention, but the protection scope of the present invention is not limited thereto. Any technician familiar with the technical field can make equivalent replacements or changes according to the technical scheme and inventive concept of the present invention within the technical scope disclosed by the present invention, which should be covered by the protection scope of the present invention.

[0202] References:

[0203] [1]IBRAHIM MS,MURALIDHARAN S,DENG Z,et al.A hierarchical deeptemporal model for group activity recognition[C] / / Proceedings of the IEEEconference on computer vision and pattern recognition, 2016:1971-1980.

[0204] [2]WU J,WANG L,WANG L,et al.Learning actor relation graphs for groupactivity recognition[C] / / Proceedings of the IEEE / CVF Conference on computervision and pattern recognition,2019:9964-9974.

[0205] [3]QI M,WANG Y,QIN J,et al.StagNet:An attentive semantic RNN forgroup activity and individual action recognition[J].IEEE Transactions onCircuits and Systems for Video Technology,2019,30(2):549-565.

[0206] [4]GAVRILYUK K,SANFORD R,JAVAN M,et al.Actor-transformers for groupactivity recognition[C] / / Proceedings of the IEEE / CVF conference on computervision and pattern recognition,2020:839-848.

[0207] [5]YAN R,TANG J,SHU X,et al.Participation-contributed temporaldynamic model for group activity recognition[C] / / Proceedings of the 26th ACMinternational conference on Multimedia,2018:1292-1300.

[0208] [6]HAN M,ZHANG D J,WANG Y,et al.Dual-AI:Dual-path actor interactionlearning for group activity recognition[C] / / Proceedings of the IEEE / CVFconference on computer vision and pattern recognition,2022:2990-2999.

[0209] [7]YUAN H,NI D,WANG M.Spatio-temporal dynamic inference network forgroup activity recognition[C] / / Proceedings of the IEEE / CVF InternationalConference on Computer Vision,2021:7476-7485.

[0210] [8]YUAN H,NI D.Learning visual context for group activity recognition[C] / / Proceedings of the AAAI Conference on Artificial Intelligence,2021:3261-3269.

[0211] [9]LI X,CHOO CHUAH M.Sbgar:Semantics based group activity recognition[C] / / Proceedings of the IEEE international conference on computer vision,2017:2876-2885.

[0212]

[10] YAN R,SHU X,YUAN C,et al.Position-aware participation-contributedtemporal dynamic model for group activity recognition[J].2021,33(12):7574-7588.

[0213]

[11] YAN R,XIE L,TANG J,et al.HiGCIN:Hierarchical graph-based crossinference network for group activity recognition[J].IEEE transactions onpattern analysis and machine intelligence,2020,45(6):6955-6968.

[0214]

[12] LU L,LU Y,WANG S.Learning multi-level interaction relations andfeature representations for group activity recognition[C] / / MultiMediaModeling:27th International Conference,MMM 2021,Prague,Czech Republic,June22-24,2021,Proceedings,Part I 27,2021:617-628.

[0215]

[13] AZAR S M,ATIGH M G,NICKABADI A,et al.Convolutional relationalmachine for group activity recognition[C] / / Proceedings of the IEEE / CVFConference on Computer Vision and Pattern Recognition,2019:7892-7901.

[0216]

[14] HU G,CUI B,HE Y,et al.Progressive relation learning for groupactivity recognition[C] / / Proceedings of the IEEE / CVF Conference on ComputerVision and Pattern Recognition,2020:980-989.

[0217]

[15] ZHAI X,HU Z,YANG D,et al.Spatial temporal network for image andskeleton based group activity recognition[C] / / Proceedings of the AsianConference on Computer Vision,2022:20-38.

[0218]

[16] ZHU X,ZHOU Y,WANG D,et al.MLST-Former:Multi-level Spatial-Temporal Transformer for Group Activity Recognition[J].IEEE Transactions onCircuits and Systems for Video Technology,2022,33(7):3383-3397.

Claims

1. A group behavior recognition method based on dynamic and static features and multiple interactive networks, characterized in that: The following steps are involved: (1) Preprocessing the original video data and skeleton node data; (2) Inception-v3 is used to extract multi-scale features from the preprocessed video data, and then RoIAlign is used to extract the static features of all participants in each frame; (3) The preprocessed skeleton node data is sequentially input into the spatial graph convolutional network and the temporal graph convolutional network to obtain the spatiotemporal motion features of the internal joints of the human body. The spatiotemporal motion features of the internal joints of the human body are then input into the TemporalAttention module to extract the overall dynamic features of the participant. (4) After changing the dimension of the dynamic features of the participants as a whole so that their dimension is the same as the static feature dimension, the static feature is merged with the dynamic features of the participants as a whole after the dimension change to obtain the dynamic and static features; (5) Use multiple coarse-grained key person extraction modules to extract information from multiple key persons, and then use multiple fine-grained person information restoration modules to restore information for all key persons. Add a GCN layer after the multiple fine-grained person information restoration modules to infer and calculate the interaction features of all participants after restoration; (6) In the fine-grained reconstruction module and the coarse-grained key person extraction module, a Maxpool layer is added after each GCN layer that integrates the participant features, and the group features output by multiple maximum pooling layers are added and fused together to form the group-level features of the character interaction; the dynamic and static features in step (4) are also connected to a Maxpool layer to aggregate into group-level features; the group-level features of the character interaction are added and fused with the group-level features to obtain multi-level character features; (7) After transposing the dimensions of the dynamic and static features, Sptial-Transformer is used to model the spatial context of all frames, and then Temporal-Transformer is used to model the temporal context of all frames to obtain complex spatiotemporal interaction features; (8) Integrate multi-level character features with complex spatiotemporal interaction features to predict the probability of occurrence of each category of group behavior Using dynamic feature X D Predict the probability of each category occurring Add the probability results of the two predictions and select the highest result P.

2. The method for identifying group behavior based on dynamic and static features and multiple interactive networks according to claim 1 is characterized in that: The preprocessing in step (1) includes: For a video input S, it is divided into K time blocks of equal time {S1,S2,...,S k }, the number of video frames in each time block is X t ,t=1,2,…,T, there are N bounding boxes for each frame Get image sequence Where 3 represents the number of channels, T represents T frames are extracted each time, H and W represent the height and width of the video frame respectively, and the input data is normalized; The skeleton node data is obtained by HRNet and is defined as Among them, N represents that there are N participants in each frame, 3 represents that the skeleton node data has three dimensions, namely the X coordinate and Y coordinate of the node and the confidence of the skeleton node, N joint Each participant has N joint A skeleton node is used to normalize the input data.

3. The method for identifying group behavior based on dynamic and static features and multiple interactive networks according to claim 1 is characterized in that: The process of extracting the dynamic features of the participants as a whole is as follows: First, construct the spatial graph G of each participant at frame t t =(V t ,E t ), where V t = {v ti ∣v ti ,i=1,...,N joint } represents the participant joint node, and the edge set E t ={e ij |i,j=1,...,N joint ,i≠j} represents the joint node v ti and v tj The relationship between Represents the characteristics of each node, where N joint represents the number of nodes, C' represents the node attribute dimension. When constructing the adjacency matrix A, it is noted that the range of movement of human joints is limited, and only node v is considered. ti With its neighborhood set N(v ti )={v tj ∣d(v ti ,v tj )≤1,j≠i}; where d(v ti ,v tj ) represents the joint node v ti and v tj The distance between Input into the spatial graph convolutional network to obtain the spatial features X of the upper joint points inside the human body S And the updated adjacency matrix A S , the formula is as follows: Then, X S Put it into the time graph convolutional network to capture the spatiotemporal motion features X of the same joint point in different frames ST , create a time graph G based on the i-th node in different frames i =(V i ,E i ), where V i = {v ti |t=1,2,...,T},E i ={e tt′ |t,t′=1,2,…,T,t′≠t}, similar to S-GCN which only considers the neighborhood of the joint point, only considers the three frames around the t-th frame, and the formula is as follows: Secondly, the spatiotemporal motion characteristics of the joints inside the human body are ST Put it into the Temporal Attention module to extract the dynamic features of the whole participant. X ST ←Temporal Attention(X ST )(4)。 4. The method for identifying group behavior based on dynamic and static features and multiple interactive networks according to claim 1 is characterized in that: Multiple coarse-grained key person extraction modules perform information extraction on multiple key persons: First, each participant node in the graph corresponds to a set of Where N is the number of participants, is the dynamic and static features of the ith participant, i.e., the dynamic and static features X in step (4) ID The i-th participant characteristics in ; The position feature of the ith participant is represented by the center coordinates; the dynamic and static features and the position features are fused into the participant features that are finally input into GCN; after the initial participant relationship is established, GCN is used to further reason about the relationship, and the original participant relationship X A The participant characteristics X are obtained through formula (7): B ; in, A is the adjacency matrix of the input graph, I is the identity matrix, yes , W is the learned weight matrix, and σ(·) is the activation function RELU; Then, in order to pay more attention to the interactions between key participants and extract key participant information as the consistency constraint information of multiple interactions into the spatiotemporal interaction, the SAGpool method is used to extract coarse-grained key person information; the key to SAGpool is that it uses GCN to provide a self-attention score Score; Score represents the importance of different participants in each frame, and the calculation formula is as follows: Among them, Score∈R N×1 is the score obtained by GCN; select the top k nodes with the largest scores to form the final character selection mask Score mask ,in, The pooling ratio λ∈(0,1] is a hyperparameter that determines the number of nodes to be retained; idx=rank(Score,k), the function rank returns the index of the top K largest scores in Score, idx It is an index operation, Score mask =Score idx ; Finally, according to the obtained Score mask , construct a coarser-grained relationship graph, and calculate the node feature X according to the character selection mask (l) And update the feature matrix X (l+1) and the adjacency matrix A (l+1) ,in, X idx,: is the row-wise indexed feature matrix, X (l+1) =X (l) ⊙Score mask , A idx,idx is the adjacency matrix indexed in row and column directions, and ⊙ denotes the broadcast element-wise product.

5. The method for identifying group behavior based on dynamic and static features and multiple interactive networks according to claim 4 is characterized in that: In the process of extracting information from multiple key figures, three coarse-grained key figure extraction modules are stacked. The key figure features obtained by the last module are 6. The method for identifying group behavior based on dynamic and static features and multiple interactive networks according to claim 1 is characterized in that: The process of restoring information of all key figures using multiple fine-grained character information restoration modules is as follows: First, the Unpool layer in the fine-grained character information restoration module puts the key nodes back to their original positions in the fine-grained relationship graph according to the indexes recorded by the key character extraction module, thereby restoring the fine-grained relationship graph; the GCN layer captures the interactions between the restored nodes; Then, a jump connection is added between the SAGpool layer of the coarse-grained key character extraction module and the Unpool layer of the fine-grained character information restoration module to better reconstruct the fine-grained character relationship graph and transfer the topological structure and deep features of the relationship graph between the two modules. The specific formula is as follows: Where idx∈Z *k is the index of k key nodes, X (l) is the feature matrix of the first layer; X (l) The features X obtained by the three key character information extraction modules kv ; Represents the initial feature matrix when the Unpool layer in the fine-grained character information restoration module reconstructs the character relationship, and its value is 0; the function reconstruct converts X according to the corresponding index recorded in idx (l) The row vectors in are assigned to In, X (l+1) Represents the reconstructed fine-grained character relationship graph. The row vector of the record index in idx will be X (l) The row vectors in are updated, while the other row vectors remain 0; Finally, as the information between the Unpool layers propagates, the last layer restores the feature information of all characters.

7. The method for identifying group behavior based on dynamic and static features and multiple interactive networks according to claim 6 is characterized in that: The number of fine-grained character information restoration modules is consistent with the number of coarse-grained key character extraction modules.

Images