A group activity recognition method and system based on action guidance

Abstract

This invention relates to the field of computer vision technology and discloses a method and system for identifying group activities based on action guidance. This method calculates action-level features by aggregating individual-level features of action subjects using an action-centric aggregation strategy. It then constructs action dependencies based on these features, and finally uses these dependencies to identify group activities. This invention addresses the low accuracy of group activity recognition, a problem inherent in existing technologies.

Description

Technical Field

[0001] The present invention relates to the field of computer vision technology, and in particular to a method and system for identifying group activities based on action guidance. Background Art

[0002] Group activity recognition aims to understand the overall activities in multi-person scenarios and has wide applications in video surveillance, social behavior understanding, and sports video analysis. Compared to individual action recognition tasks (e.g., jumping or skiing), group activity recognition is a more challenging task (e.g., queuing or talking) as it not only collects individual actions but also explores the interactions between multiple people.

[0003] Currently, there are various solutions dedicated to identifying group activities, which can be summarized as the following process. First, individual-level features of randomly selected video frames in the video clip are extracted based on backbone networks (such as ResNet18 and Inception-v3). Secondly, the spatiotemporal relationships between multiple people are explored through graph convolutional networks (GCN), recurrent neural networks (RNN) or the recent transformer. Finally, an overall feature vector is generated by fusing features to represent group activities. However, most existing methods are limited to coarse individual-level features and ignore some specific relationships. Although some deep learning-based methods have achieved significant success, group activity recognition still faces great difficulties in learning spatial and temporal relationships due to complex motion changes and action transitions. In fact, only a few people and their interactions or actions will have an important impact on inferring group activities. In order to focus on these related people, CCG-LSTM quantifies the contribution to a certain motion through spatiotemporal context correlation and global context constraints. P 2 CTDM defines key participants as those whose movements are stable throughout the entire process or whose movements change significantly at a specific moment. It then learns visual and spatial features by gradually capturing both long-term and instantaneous spatiotemporal actions. However, these methods are limited to modeling person-level relationships. Further exploration of the underlying relationships between people is necessary for reasoning about group activities.

[0004] Graph neural network-based methods have gained increasing attention due to their advantages in relational reasoning. Nodes represent individual-level features, and edges represent the relationship between two people. To capture the context of interactions between specific people, researchers have proposed a dynamic reasoning network to jointly model global-level relations. Through graph convolution operations, these methods have the natural and flexible ability to capture rich relational information in an end-to-end manner. However, distinguishing confusable group activities based solely on individual-level features and their spatial information is difficult, without comprehensively considering temporal evolution. Inspired by the progress made in visual transformers, encoder-decoder architectures have been used to refine individual-level features. ActorFormer first introduced transformers by leveraging the encoder structure to capture group-related representations. To holistically explore spatiotemporal information, GroupFormer augments individual and group features with a clustered spatiotemporal transformer. While these efforts strengthen spatiotemporal relationships, they do not explicitly model semantic action dependencies between all individuals in the group. Furthermore, individual-level features are directly fed into the self-attention module without any refinement, making these methods difficult to handle unbalanced spatiotemporal interactions. In summary, existing technologies have shortcomings such as the gap between local and global feature representation, the lack of refinement of individual-level features, and the lack of clear modeling of semantic action dependencies, which lead to problems such as low accuracy in group activity recognition. Summary of the Invention

[0005] To overcome the deficiencies of the prior art, the present invention provides a method and system for group activity recognition based on action guidance, which solves the problems of low accuracy in group activity recognition in the prior art.

[0006] The technical solution adopted by the present invention to solve the above problems is:

[0007] An action-guided group activity recognition method calculates action-level features by aggregating individual-level features of action subjects through an action-centered aggregation strategy, then constructs action dependency relationships based on the action-level features, and finally uses the action dependency relationships to perform group activity recognition.

[0008] As a preferred technical solution, the following steps are included:

[0009] S1, individual-level feature extraction: Extract features from the RGB video frame to obtain individual-level features of each action subject in the RGB video frame; individual-level features refer to appearance features used to characterize the action subject in the RGB video frame;

[0010] S2, action-level feature calculation: Aggregate the individual-level features obtained in step S1 to calculate action-level features; action-level features refer to features used to characterize actions in RGB video frames;

[0011] S3, action-level feature learning: Use an encoder with a self-attention mechanism to perform self-attention calculation on the action-level features obtained in step S2 to obtain refined action-level features;

[0012] S4, action dependency construction: Using the refined action-level features obtained in step S3, a semantic action graph is constructed, and then action dependencies are constructed based on the semantic action graph;

[0013] S5, group activity recognition: Using the action dependency relationship constructed in step S4, group activity recognition is performed on the RGB video frame, and the group activity category is output; wherein the group activity category refers to the action category that is globally determined for all action subjects in the RGB video frame.

[0014] As a preferred technical solution, step S1 includes the following steps:

[0015] S11, receiving T frames of RGB video frames, wherein each frame has N bounding boxes of action subjects, T represents the number of RGB video frames, and N represents the number of action subjects in each frame;

[0016] S12, extracting image features of RGB video frames using the backbone network;

[0017] S13, extract individual-level features from image features, individual-level features are represented as X∈R T×N×D , Among them, X represents individual-level features, D represents the dimension of image features extracted by the backbone network, R represents the vector space, i represents the number of the action subject, 1≤i≤T×N, x i is the individual-level feature of the i-th action subject.

[0018] As a preferred technical solution, step S2 includes the following steps:

[0019] S21, assign the individual-level features in each frame to all action words, and calculate the residual between the individual-level features and the action words;

[0020] S22, input the individual-level features in each frame into LP to obtain linearized individual-level features, and then input N linearized individual-level features into Softmax to obtain the soft assigned weight corresponding to each individual-level feature;

[0021] S23, multiplying the residual between each individual-level feature and the action obtained in step S21 by the soft assigned weight corresponding to each individual-level feature obtained in step S22 to obtain the action feature component corresponding to each individual-level feature;

[0022] S24, adding the action feature components corresponding to each individual-level feature obtained in step S23, and superimposing the action-level features corresponding to K action words to obtain the action-level feature of the frame x′ k The calculation formula is:

[0023]

[0024] Where K represents the number of action words, k represents the number of action words, and x′ k represents the action-level feature corresponding to the k-th action word in the frame, i and j represent the numbers of any two action subjects, and w j Indicates that x j Projected as a scalar learnable weight, b j Indicates that x j The bias projected as a scalar, A K represents the kth action word, w i Represents x i Projected to a scalar, the learnable weights, b i Indicates that x i The bias projected as a scalar, x j represents the individual-level features of the j-th action subject.

[0025] As a preferred technical solution, step S3 includes the following steps:

[0026] S31, uses a temporal encoder with self-attention mechanism and a spatial encoder to calculate the query, key and value respectively:

[0027] Among them, the temporal encoder is used to extract the action-level features X′∈R of T frames. T×K×D Compute the query, key, and value of the action-level features in n self-attention heads:

[0028] Use the spatial encoder to extract the action-level feature X′ of the t-th frame t ∈R K×D , calculate the query, key and value of action-level features in n self-attention heads:

[0029]

[0030]

[0031] Where n represents the sequence number of the attention head, 1≤n≤S, S represents the number of attention heads, t represents the sequence number of the frame, 1≤t≤T, represents the query of the nth attention head of the temporal encoder, represents the query weight of the nth attention head of the temporal encoder, represents the key of the nth attention head of the temporal encoder, represents the key weight of the nth attention head of the temporal encoder, represents the value of the nth attention head of the temporal encoder, represents the value weight of the nth attention head, represents the query of the t-th frame on the n-th attention head in the spatial encoder, represents the query weight of the nth attention head in the spatial encoder, represents the key of the t-th frame on the n-th attention head in the spatial encoder, represents the query weight of the nth attention head in the spatial encoder, represents the value of the t-th frame at the n-th attention head in the spatial encoder, represents the value weight of the nth attention head in the spatial encoder;

[0032] S32, calculate the nth attention learned by the temporal encoder and spatial encoder, the calculation formula is:

[0033]

[0034]

[0035] in, represents the nth attention learned by the temporal encoder, Transpose represents the matrix transposition operation, represents the nth attention of the tth frame learned by the spatial encoder;

[0036] S33 calculates the attention learned by the temporal encoder and the spatial encoder. The calculation formula is:

[0037]

[0038]

[0039] Among them, A Temporal represents the attention learned by the temporal encoder, Concat represents the matrix concatenation operation, and W Temporal represents the temporal encoder weight, represents the attention of the t-th frame learned by the spatial encoder; W Spatial represents the spatial encoder weight;

[0040] S34, calculates the action-level features after the temporal encoder and spatial encoder are refined;

[0041]

[0042]

[0043]

[0044] in, represents the action-level features after the temporal encoder refinement, LN represents the layer normalization operation, FW represents the combination of linear projection and ReLU operations, represents the action-level features of the t-th frame after refinement of the spatial encoder, It represents the feature superposition result after T frame refinement;

[0045] S35, the refined action-level features are obtained by averaging the time and space features along the time dimension.

[0046]

[0047] in, express The action-level features of the t-th frame in express The action-level features of the t-th frame in .

[0048] As a preferred technical solution, S4 uses semantic action graphs to explicitly model high-level action dependencies, as shown below:

[0049] G={G m,o ∈R 1 |m,o=1,...,K};

[0050]

[0051] Where G represents the graph of pairwise action-level dependencies, G∈R K×K , G m.o Represents feature pairs The relationship value, R 1 represents a 1-dimensional vector space, m and o represent the numbers of any two action words, K represents the number of action words, Represents the refined action-level features The feature pairs in represents the refined action-level features, represents the refined action-level features, exp() represents the exponential function with constant e as the base, and φ represents the trainable weight. Represents feature pairs The action relationship value of b represents the bias.

[0052] As a preferred technical solution, in S4, the output features are obtained by fusing the refined action-level features and the graph-based action dependency features, as shown below:

[0053]

[0054] in, Represents group-level features, Relu represents nonlinear activation function, W G ∈R D×D represents the learnable graph weight matrix.

[0055] As a preferred technical solution, in S4, the loss function L activity And the loss function L action The two constrained classifiers are used to predict group activity confidence and video-level action confidence respectively:

[0056] L=L activity (y1,p1)+λL action (y2,p2);

[0057] Among them, L represents the total loss, L activity (y1, p1) represents the group activity loss, y1 represents the true label of the group activity, p1 represents the confidence of the group activity, λ represents the hyperparameter for balancing the two losses, and L action (y2, p2) represents the video-level action loss, y2 represents the true label of the video-level action, p2 represents the action word confidence, group activity confidence refers to the action category and its probability of global judgment of the actions of all action subjects in the RGB video frame, and action word confidence refers to the action category and its probability of a certain action subject in the RGB video frame.

[0058] As a preferred technical solution, the loss function is calculated as follows:

[0059] L activity (y1,p1)=-y1log(p1);

[0060]

[0061] in, represents the true label of the k-th video-level action, represents the confidence of the k-th action word.

[0062] A group activity recognition system based on action guidance, used to implement the above-mentioned group activity recognition method based on action guidance, includes the following modules connected in sequence:

[0063] Individual-level feature extraction module: used to extract features from RGB video frames and obtain individual-level features of each action subject in the RGB video frame; individual-level features refer to the appearance features used to characterize the action subject in the RGB video frame;

[0064] Action-level feature calculation module: used to aggregate individual-level features and calculate action-level features; action-level features refer to features used to characterize actions in RGB video frames;

[0065] Action-level feature learning module: used to perform self-attention calculation on action-level features using an encoder with a self-attention mechanism to obtain refined action-level features;

[0066] Action dependency building module: used to build a semantic action graph using refined action-level features, and then build action dependencies based on the semantic action graph;

[0067] Group activity recognition module: Used to identify group activities in RGB video frames using action dependencies and output group activity categories; where the group activity category refers to the action category that is globally determined for all action subjects in the RGB video frame.

[0068] Compared with the prior art, the present invention has the following beneficial effects:

[0069] (1) This paper adopts an action-guided spatiotemporal transformer to model the unbalanced spatiotemporal relationships for group action recognition;

[0070] (2) The present invention adopts an action-centered aggregation strategy (A 2 S), used to bridge the gap between individual-level features and group-level representations, and learn action-level features under the supervision of global action information;

[0071] (3) The present invention adopts a dual-branch transformer (DT) to refine action-level features, which is built by two self-attention encoder modules;

[0072] (4) The present invention explicitly constructs a semantic action graph (SAG) to model dynamic action dependencies through structural action embedding. BRIEF DESCRIPTION OF THE DRAWINGS

[0073] Figure 1 A schematic diagram of the steps of a group activity recognition method based on action guidance according to the present invention;

[0074] Figure 2 This is an example diagram of group activities for "right spike" and "right set";

[0075] Figure 3 This is the overall architecture diagram of the present invention;

[0076] Figure 4 To obtain the kth potential action A through the action-centric aggregation module k Schematic diagram of action-level features;

[0077] Figure 5 This is a flow chart of the algorithm of the present invention. DETAILED DESCRIPTION

[0078] The present invention will be further described in detail below with reference to the embodiments and the accompanying drawings, but the embodiments of the present invention are not limited thereto.

[0079] Example 1

[0080] like Figures 1 to 5 As shown, this application relates to the field of computer vision technology, specifically to group activity recognition methods, model training methods, and deployment. Group activity recognition tasks require not only identifying the individual movement patterns of each person, but also analyzing the relationships between all individuals participating in the group activity in the video, and further combining scene information to determine the behavior category of the group activity. In this invention, the action subject refers to the individual in motion, and in this embodiment, athletes are used as an example.

[0081] The present invention solves and innovates the following problems in the prior art:

[0082] 1. There is a gap between local individual-level features and global group feature representations;

[0083] Figure 2 This is an example of a group activity for "right spike" and "right set." Spiking and blocking are closely related to "right spike." Meanwhile, set and movement contribute to inferring "right set." Only a few individuals performing specific actions are more critical than others standing on the scene performing the corresponding activity, leading to an imbalance in spatiotemporal interactions. Intuitively, the intermediate action-level features are more closely related to group activities than individual-level features. The interactions surrounding these actions are more dominant than those surrounding other actions, further providing meaningful clues for inferring these two group activities.

[0084] To better illustrate this, some participants who perform special actions and their interactions with other participants are crucial for inferring group activities. Figure 2In this paper, we use two different group activities, "spiking right" and "passing right," as examples. For "spiking right," two actions (spiking and blocking) are more critical than the others, while two other actions (passing and moving) are related to "passing right." Although these specific actions offer a natural and flexible advantage in distinguishing the two group activities, an inevitable gap remains between local individual-level features and global group feature representations, which few methods have explored in detail. Intermediate action-level features, which are more closely related to group activities than individual-level features, can be used to bridge this gap. Therefore, action-centered aggregation of individual-level features is necessary for learning high-level spatiotemporal relationships. Furthermore, the interactive relationships surrounding these actions are more dominant than those surrounding other actions, which can provide meaningful clues for inferring group activities. Based on these findings, we propose an action-centered aggregation strategy and model spatiotemporal action dependencies for group action recognition. Therefore, we propose an action-guided spatiotemporal transformer that captures interactive relationships by aggregating individual-level features and modeling relationships between actions. We represent latent action words by learning a few action-related cluster centers, and all individual-level features in each frame are assigned to the corresponding action word. Then, under the supervision of global action information, action-level features are aggregated by weighting the sum of the residuals of each action word. In addition, the present invention also models the related action words in a weakly supervised manner.

[0085] 2. The individual-level features input to the self-attention module are not refined;

[0086] To address the problem that individual-level features are not refined, this paper proposes a dual-branch spatiotemporal transformer to refine inter-frame and intra-frame action-level features, in which two encoders with self-attention mechanisms select important tags by capturing local spatial information and global temporal information respectively.

[0087] 3. Failure to explicitly model semantic action dependencies between all people in the group;

[0088] We explicitly design semantic action graphs to model dynamic action dependencies for group activity recognition. We then fuse these important cues to obtain a comprehensive group representation while only requiring video-level action labels.

[0089] Specifically, the following features are included:

[0090] 1. Overall architecture:

[0091] Figure 3 The overall architecture diagram of the present invention is as follows: Given a sequence segment, RoIAlign extracts individual-level features based on the backbone network. First, an action-centric aggregation strategy (A 2S), under the supervision of global action information, action-level features are learned by weighting the residual sum of each action word. Then, the inter-frame and intra-frame action-level features are fed into a two-branch transformer with two encoders to refine important feature markers. In addition, the semantic action graph is explicitly used to model dynamic action dependencies. Finally, the fused features are combined and fed into a softmax layer to predict (i.e., recognize) group activities.

[0092] The present invention receives a sequence of T frames, each with N bounding boxes of people, which are fed into RoIAlign to extract individual-level features based on a predefined backbone network. The present invention uses ResNet18 and Inception-v3 as the backbone network. The individual-level features can be expressed as X∈R T×N×D , where T, N, D represent time, space, and feature dimensions respectively, R represents vector space, X represents individual-level features, and N represents the number of action subjects in each RGB video frame, i.e., the number of individual-level features. Considering the group activities of interest, the actions involved are known. First, assuming K action-related cluster centers to represent potential action words, an action-centric aggregation strategy is proposed to extract action-level features X′∈R from all people in the entire group. T×K×D , K represents the number of action words, and X′ is the action-level feature, which can be obtained by analyzing all actions involved in group activities. All individual-level features in each frame are assigned to all action words, and then the action-level features can be calculated by weighting the residual sum of each action (i.e., the difference between the individual and each action word). The action-level features are then sent to a two-branch transformer, in which two encoders with self-attention mechanisms are used to select important feature markers along the inter-frame and intra-frame directions respectively. In addition, a semantic action graph is explicitly constructed to model dynamic action dependencies, and group-level representations are obtained by fusing these important clues. Finally, following the previous method, the fused features are merged and sent to the softmax layer to predict group activities. The cross-entropy loss function is used to train the model, which can be trained in an end-to-end manner. For the inference of video clips, the final probability score of the group activity is obtained by averaging the outputs of the softmax classifier.

[0093] 2. Action-centered aggregation module:

[0094] As mentioned in the background, most recent methods follow a similar process where individual-level features are fed into various graph neural networks or transformers for relational reasoning, without any improvement. In fact, individual-level features are redundant and not all of them can have a critical impact on understanding group activities, as shown in the attached figure. Figure 2As shown in Figure 2. Although the performance of group activity recognition is constantly improving, previous methods have not carefully explored the gap between local individual-level features and global group-level representations. In this paper, learning action-level features based on original individual-level features is a key component. Inspired by ActionVLAD, the cluster centers associated with actions can be used to represent K potential action words. The basic idea is to construct an action-centered aggregation module under the global supervision of video-level action information. Specifically, individual-level features are used to represent the action-level representation of the action words. is fed into the action-centric aggregation module in a frame-by-frame manner, where x i is the individual-level feature of the i-th person. For convenience, the output of the action-centric aggregation module is x′ k represents the kth action-level feature. Assume A K represents the kth action word, x′ k It is by A K and all individual-level characteristics Calculated by aggregating residuals, that is:

[0095]

[0096] r(x j ) represents an indicator signal, indicating whether the jth person has done the kth action in the k action words. In short, if x j and A K There is a residual relationship, then r(x j )=1, otherwise r(x j )=0. It is easy to see that the above formula is not differentiable and cannot be directly trained by backpropagation. Similar to NetVLAD, for x j With A K The soft assignment weight r(x j ) is defined as:

[0097]

[0098] where w j and b j is x j Projected to a scalar, we can learn the weights and substitute them into x′ k It can be expressed as:

[0099]

[0100] The number of potential action words K can be determined by all actions involved in the group activity of interest without the need to annotate each person in a frame. Combined with the standard group activity loss L activity , cross entropy loss L actionis used to learn under the constraints of video-level action labels

[0101] Figure 4 The action-level feature diagram of the kth potential action is obtained through the action-centered aggregation module. k represents the kth action-level feature, A K represents the Kth action word, x′ k By A K and all individual-level features are calculated by aggregating the residuals.

[0102] 3. Dual-branch transformer:

[0103] Although action-centric aggregation is explicitly modeled to learn action-level features, it is not aware of positional correlations and may not capture temporal relationships. Considering the advantages of the self-attention mechanism in capturing long-term relationships, the present invention introduces a dual-branch transformer to capture spatiotemporal contextual information. It includes two encoders to refine inter-frame and intra-frame features. Given the action-level features X′ of T frames, a temporal encoder is adopted to model the temporal evolution of each potential action on the frame by taking all action-level features as labels. In the self-attention module, there are a total of S heads. The projection of the nth query, key, and value as the encoded features It is generated by the linear projection layer, and the formula is as follows:

[0104]

[0105]

[0106]

[0107] Where n represents the sequence number of the attention head, 1≤n≤S, S represents the number of attention heads, t represents the t-th frame in the spatial encoder, 1≤t≤T, represents the query of the nth attention head of the temporal encoder, represents the query weight of the nth attention head of the temporal encoder, represents the key of the nth attention head of the temporal encoder, represents the key weight of the nth attention head of the temporal encoder, represents the value of the nth attention head of the temporal encoder, Represents the value weight of the nth attention head, A Temporal represents the attention learned by the temporal encoder, Concat represents the matrix concatenation operation, and W Temporal represents the temporal encoder weight,

[0108] In this way, the action-level features refined by the temporal encoder It can be obtained according to the following formula:

[0109]

[0110] Among them, LN represents the layer normalization operation, and FW represents the combination of linear projection and ReLU operations.

[0111] Another spatial encoder is used to capture the spatial relationship between multiple potential actions in each frame. For simplicity, the spatial encoder follows the operation of the temporal encoder. The difference between them is that the learned action-level features are fed into the spatial encoder in a frame-by-frame manner. Given the action-level feature X′ of the input frame t t , the feature maps of T frames are superimposed on

[0112]

[0113]

[0114]

[0115]

[0116]

[0117] in represents the query of the t-th frame on the n-th attention head in the spatial encoder, represents the query weight of the nth attention head in the spatial encoder, represents the key of the t-th frame on the n-th attention head in the spatial encoder, represents the query weight of the nth attention head in the spatial encoder, represents the value of the t-th frame at the n-th attention head in the spatial encoder, represents the value weight of the nth attention head in the spatial encoder; W represents the attention learned by the spatial encoder up to the tth frame; Spatial represents the spatial encoder weights, It represents the feature superposition result after T frame refinement.

[0118] Finally, the refined action-level features It can be obtained by averaging the refined temporal and spatial features along the time dimension, namely:

[0119]

[0120] in, express The action-level features of the t-th frame in express The action-level features of the t-th frame in .

[0121] 4. Semantic Action Graph:

[0122] Different group activities may involve some overlapping behaviors. Only the simultaneous occurrence of a few specific behaviors is conducive to reasoning about group activities. It is necessary to model the action dependencies in group activities. This paper uses semantic action graphs to explicitly model high-level action dependencies, which is a supplement to the spatiotemporal relationship based on transformers. Given a pair of refined features The feature pairs in The linear projection method is used to calculate the action relationship value Modeling. The graph of pairwise action-level dependencies can be represented as G = {G m,o ∈R 1 |m,o=1,...,K}, where G∈R K×K . Relationship value G m.n It can be expressed as:

[0123]

[0124] Where G represents the graph of pairwise action-level dependencies, G∈R K×K , G m.o Represents feature pairs The relationship value, R 1 represents a 1-dimensional vector space, m and o represent the numbers of any two action words, K represents the number of action words, Represents the refined action-level features The feature pairs in represents the refined action-level features, represents the refined action-level features, exp() represents the exponential function with constant e as the base, and φ represents the trainable weight. Represents feature pairs The action relationship value of b represents the bias.

[0125] After building the graph, a layer of graph convolutional network is used to perform relational reasoning. The output features can be obtained by fusing the refined action-level features and the graph-based action dependency features as shown below:

[0126]

[0127] in, Represents group-level features, Relu represents nonlinear activation function, W G ∈R D×D represents the learnable graph weight matrix.

[0128] For the training phase, L activity and L action The two constrained classifiers are used to predict the group activity score and the video-level action score, respectively. For these two losses, cross entropy is used to guide the optimization process:

[0129] L=L activity (y1,p1)+λL action (y2,p2)

[0130] Among them, L represents the total loss, L activity (y1, p1) represents the group activity loss, y1 represents the true label of the group activity, p1 represents the confidence of the group activity, λ represents the hyperparameter for balancing the two losses, and L action (y2, p2) represents the video-level action loss, y2 represents the true label of the video-level action, and p2 represents the video-level action confidence. The group activity confidence refers to the action category and its probability of global judgment of the actions of all action subjects in the RGB video frame, and the video-level action confidence refers to the action category and its probability of a certain action subject in the RGB video frame.

[0131]

[0132] in, represents the true label of the k-th video-level action, represents the confidence of the k-th action word.

[0133] The present invention has the following advantages:

[0134] 1. Overall effect:

[0135] The proposed action-guided spatiotemporal transformer captures interactive relationships by aggregating individual-level features and modeling action correlations, effectively overcoming the shortcomings of current related methods. It outperforms most popular methods on the Volleyball and CollectiveActivity datasets, significantly improving performance by 1.8% compared to HiGCIN, which also uses a ResNet-18 backbone network. Compared to the best Inception-v3-based method, GroupFormer, the proposed method improves MCA by 2.9%, demonstrating its superiority in learning spatiotemporal relationships.

[0136] 2. Action-centered aggregation module:

[0137] The action-centered aggregation module (A 2 S) The performance is improved by 0.45% compared with the baseline method, verifying the 2S can well learn the gap between local individual-level features and global group-level representations. In addition, the present invention adds a dual-branch transformer and semantic action graph module to ActionVLAD through experiments and finds that its performance is lower than that of the present invention, which also reflects that A 2 The action-level information captured by S is beneficial for subsequent relation learning.

[0138] 3. Dual-branch transformer:

[0139] The dual-branch transformer (DT) proposed in this paper improves the performance by 1.02% compared with the baseline method, verifying the significant effect of DT in capturing spatiotemporal contextual information by refining inter-frame and intra-frame features.

[0140] 4. Semantic Action Graph:

[0141] The proposed Semantic Action Graph (SAG) achieves a 0.60% performance improvement over the baseline method, demonstrating the capability of SAG in modeling dynamic behavioral dependencies.

[0142] In summary, the present invention adopts an action-guided spatiotemporal transformer to model the unbalanced spatiotemporal relationship of group activity recognition. 2 S) is used to bridge the gap between individual-level features and group-level representations, learning action-level features under the supervision of global action information. This paper uses a dual-branch transformer (DT) to refine action-level features, where action interactions are modeled by two self-attention encoder modules. The semantic action graph (SAG) is explicitly constructed to model dynamic action dependencies through structural action embedding.

[0143] As described above, the present invention can be preferably implemented.

[0144] All features disclosed in all embodiments in this specification, or steps in all methods or processes implicitly disclosed, except for mutually exclusive features and / or steps, can be combined and / or expanded or replaced in any manner.

[0145] The above description is merely a preferred embodiment of the present invention and does not constitute any form of limitation to the present invention. Based on the technical essence of the present invention and within the spirit and principles of the present invention, any simple modification, equivalent replacement and improvement of the above embodiment shall still fall within the scope of protection of the technical solution of the present invention.

Claims

1. A group activity recognition method based on action guidance, characterized in that: The action-centered aggregation strategy aggregates the individual-level features of the action subject to calculate the action-level features. Then, action dependencies are constructed based on the action-level features. Finally, the action dependencies are used to identify group activities. The following steps are included: S1, individual-level feature extraction: extract features from the RGB video frame to obtain individual-level features of each action subject in the RGB video frame; individual-level features refer to appearance features used to characterize the action subject in the RGB video frame; S2, action-level feature calculation: Aggregate the individual-level features obtained in step S1 to calculate action-level features; action-level features refer to features used to characterize actions in RGB video frames; S3, action-level feature learning: Use an encoder with a self-attention mechanism to perform self-attention calculation on the action-level features obtained in step S2 to obtain refined action-level features; S4, action dependency construction: Using the refined action-level features obtained in step S3, a semantic action graph is constructed, and then action dependencies are constructed based on the semantic action graph; S5, group activity recognition: using the action dependency relationship constructed in step S4, group activity recognition is performed on the RGB video frame, and the group activity category is output; wherein the group activity category refers to the action category determined globally for all action subjects in the RGB video frame; Step S2 includes the following steps: S21, assign the individual-level features in each frame to all action words, and calculate the residual between the individual-level features and the action words; S22, input the individual-level features in each frame into LP to obtain linearized individual-level features, and then input N linearized individual-level features into Softmax to obtain the soft assigned weight corresponding to each individual-level feature; S23, multiplying the residual between each individual-level feature and the action word obtained in step S21 by the soft assigned weight corresponding to each individual-level feature obtained in step S22 to obtain the action feature component corresponding to each individual-level feature; S24, adding the action feature components corresponding to each individual-level feature obtained in step S23, and superimposing the action-level features corresponding to K action words to obtain the action-level feature of the frame , The calculation formula is: ； Among them, K represents the number of action words, k represents the number of action words, represents the action-level feature corresponding to the k-th action word in the frame, i and j represent the numbers of any two action subjects, Indicates that The projected scalar learnable weights, Indicates that The bias projected as a scalar, represents the k-th action word, express projected to a scalar learnable weight, Indicates that The bias projected as a scalar, represents the individual-level features of the j-th action subject.

2. The method for identifying group activities based on action guidance according to claim 1, characterized in that: Step S1 includes the following steps: S11, accepting a video to be recognized, and randomly extracting T frames of RGB video frames from it; each frame has N bounding boxes of action subjects, T represents the number of extracted video frames, and N represents the number of action subjects in each frame; S12, extracting image features of RGB video frames using the backbone network; S13, extract individual-level features from the image features of the video frame. The individual-level features are expressed as , ; Where X represents individual-level features, D represents the feature dimension extracted by the backbone network, R represents the vector space, i represents the number of the action subject, 1≤i≤T×N, is the individual-level feature of the i-th action subject.

3. The method for identifying group activities based on action guidance according to claim 1, characterized in that: Step S3 includes the following steps: S31, uses a temporal encoder with self-attention mechanism and a spatial encoder to calculate the query, key and value respectively: Among them, the time encoder is used to calculate the action level features of T frames Compute the query, key, and value of the action-level features in n self-attention heads: Use the spatial encoder to extract the action-level features of the t-th frame , calculate the query, key and value of action-level features in n self-attention heads: ； ； Where n represents the sequence number of the attention head, 1≤n≤S, S represents the number of attention heads, t represents the sequence number of the frame, 1≤t≤T, represents the query of the nth attention head of the temporal encoder, represents the query weight of the nth attention head of the temporal encoder, represents the key of the nth attention head of the temporal encoder, represents the key weight of the nth attention head of the temporal encoder, represents the value of the nth attention head of the temporal encoder, represents the value weight of the nth attention head, represents the query of the t-th frame on the n-th attention head in the spatial encoder, represents the query weight of the nth attention head in the spatial encoder, represents the key of the t-th frame on the n-th attention head in the spatial encoder, represents the query weight of the nth attention head in the spatial encoder, represents the value of the t-th frame at the n-th attention head in the spatial encoder, represents the value weight of the nth attention head in the spatial encoder; S32, calculate the nth attention learned by the temporal encoder and spatial encoder, the calculation formula is: ； ； in, represents the nth attention learned by the temporal encoder, represents the matrix transpose operation, represents the nth attention of the tth frame learned by the spatial encoder; S33 calculates the attention learned by the temporal encoder and the spatial encoder. The calculation formula is: ； ； in, represents the attention learned by the temporal encoder, represents the matrix concatenation operation, represents the temporal encoder weight, represents the attention of the t-th frame learned by the spatial encoder; represents the spatial encoder weight; S34, calculates the action-level features after the temporal encoder and spatial encoder are refined; ； ； ； in, represents the action-level features after the temporal encoder refinement, Representation layer normalization operation, represents the combination of linear projection and ReLU operations, represents the action-level features of the t-th frame after refinement of the spatial encoder, It represents the feature superposition result after T frame refinement; S35, the refined action-level features are obtained by averaging the time and space features along the time dimension. ; ； in, express The action-level features of the t-th frame in express The action-level features of the t-th frame in .

4. The method for identifying group activities based on action guidance according to claim 3, characterized in that: In S4, semantic action graphs are used to explicitly model high-level action dependencies, which are expressed as follows: ； ； in, A graph representing pairwise action-level dependencies, , Represents feature pairs The relationship value of represents a 1-dimensional vector space, 、 represents the numbers of any two action words, Indicates the number of action words, Represents the refined action-level features The feature pairs in represents the refined action-level features, represents the refined action-level features, exp() represents the exponential function with constant e as the base, represents the trainable weights, Represents feature pairs The action relationship value of b represents the bias.

5. The method for identifying group activities based on action guidance according to claim 4, characterized in that: In S4, the output features are obtained by fusing the refined action-level features and the graph-based action dependency features, as shown below: ； in, represents the group-level features, represents a nonlinear activation function, represents the learnable graph weight matrix.

6. The method for identifying group activities based on action guidance according to claim 4 or 5, characterized in that: In S4, the loss function and loss function The two constrained classifiers are used to predict group activity confidence and video-level action confidence respectively: ； in, represents the total loss, Indicates the loss of group activity, represents the true label of the group activity, represents the confidence of group activity, represents the hyperparameter that balances the two losses, represents the video-level motion loss, represents the true label of the video-level action, Represents the confidence of the action word. The group activity confidence refers to the action category and its probability for the global judgment of the actions of all action subjects in the RGB video frame. The action word confidence refers to the action category and its probability of a certain action subject in the RGB video frame.

7. The method for identifying group activities based on action guidance according to claim 6, characterized in that: The loss function is calculated as follows: ； ； in, represents the true label of the k-th video-level action, represents the confidence of the k-th action word.

8. A group activity recognition system based on action guidance, characterized in that: A method for group activity recognition based on action guidance according to any one of claims 1 to 7 is implemented, comprising the following modules connected in sequence: Individual-level feature extraction module: used to extract features from RGB video frames and obtain individual-level features of each action subject in the RGB video frame; individual-level features refer to the appearance features used to characterize the action subject in the RGB video frame; Action-level feature calculation module: used to aggregate individual-level features and calculate action-level features; action-level features refer to features used to characterize actions in RGB video frames; Action-level feature learning module: used to perform self-attention calculation on action-level features using an encoder with a self-attention mechanism to obtain refined action-level features; Action dependency building module: This module uses the refined action-level features to build a semantic action graph, and then builds action dependencies based on the semantic action graph. Group activity recognition module: Used to identify group activities in RGB video frames using action dependencies and output group activity categories; where the group activity category refers to the action category that is globally determined for all action subjects in the RGB video frame.

Images