A coarse-grained-fine-grained nested learning-based group activity recognition method
By using a coarse-grained-fine-grained nested learning method, visual block features are gradually located and refined, solving the problem of time-consuming and inflexible visual feature extraction in existing technologies, and achieving efficient and accurate group activity recognition.
Patent Information
- Application Number
- CN202310624241.2
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2023-05-30
- Publication Date
- 2026-01-13
- Estimated Expiration
- 2043-05-30
AI Technical Summary
Existing methods for identifying group activities rely on fine-grained labels, which makes extracting visual features time-consuming and inflexible, and unable to accurately capture key visual features, especially under weak supervision conditions where it is difficult to effectively extract effective visual features.
We adopt a coarse-grained-fine-grained nested learning approach. Through nested interactive learning, coarse-grained spatial localization, and fine-grained spatiotemporal selection, we gradually locate and refine visual block features, reduce dependence on detectors, adaptively select key visual parts, and deeply mine their relationships.
It improves the accuracy of group activity identification under weak supervision, can efficiently analyze group activity categories, reduces reliance on individual location and action category labels, and enhances the flexibility and accuracy of analysis.
Smart Images
Figure CN116630892B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to a method for recognizing group activity in the field of computer vision, and particularly to a method for recognizing group activity without a detector. Background Technology
[0002] Group activity recognition is a fundamental yet challenging task in computer vision, with wide applications in surveillance, sports video analysis, and social role understanding. Faced with the redundant visual information in multi-person scenes, existing methods rely on fine-grained labels, such as individual location and action tags, to extract effective individual features or characteristics. Specifically, these methods use single location tags and RoIAlign to extract individual features, capturing interaction information and spatiotemporal changes between individuals, and then generating video representations that reflect both interaction and scene information. However, obtaining these fine-grained features is very time-consuming, which is a significant challenge in group activity recognition.
[0003] Therefore, to reduce reliance on fine-grained labels and improve the applicability of group activity recognition, researchers have begun exploring weakly supervised group activity recognition methods by using only video-level activity labels during both the training and testing phases. The main challenge of weakly supervised group activity recognition is how to extract effective key visual features without any additional fine-grained personal supervision information. Existing solutions extract visual features in the following ways: (a) Person features. Off-the-shelf object detectors are used to generate person location nominations, and then person features are extracted. However, this heavily relies on the quality of existing detectors and lacks flexibility. (b) Mesh features. A Transformer is used to learn a set of token embeddings from noisy mesh features with learnable weights, and then these are aggregated into the overall video representation. However, this approach encodes all local visual information equally and does not delve into key visual content highly relevant to the activity. Therefore, how to capture key visual features flexibly and accurately is a problem worthy of attention. Summary of the Invention
[0004] Purpose of the invention: To address the aforementioned existing technologies, this invention proposes a detector-free group activity recognition method based on coarse-grained and fine-grained nested learning, which effectively improves the efficiency of group activity recognition under weak supervision.
[0005] Technical solution: A group activity recognition method based on coarse-grained and fine-grained nested learning, comprising the following steps:
[0006] S1 Nested Interactive Learning: The T-frame RGB images are cropped and then fed into the skeleton network to extract visual features. Each feature is treated as a local token. A learnable global token is defined for each frame and concatenated with the local tokens to update the visual features. The visual features are then fed into the nested interactive network to capture spatiotemporal global information and obtain global features.
[0007] S2 coarse-grained spatial localization: The spatial visual weight of each local token is represented by the similarity between the local token and the global token. Then, local tokens with high similarity are roughly selected and mapped to the original image, which is to roughly locate the key visual blocks.
[0008] S3 Fine-grained spatiotemporal selection: The local block features obtained in step S2 are refined using a block fusion module and a cross-attention mechanism;
[0009] S4 Group Activity Prediction: Input the video representation obtained in step S3 into the classifier for classification.
[0010] Furthermore, in step S1, a frame of image contains N local tokens, which are concatenated with a global token to obtain new visual features. Where C is the feature dimension; the step of inputting visual features into a nested interactor to capture spatiotemporal global information includes the following steps:
[0011] S11: Visual feature X g A learnable spatial location encoding is added, and then a spatial attention mechanism is used to explore the spatial interaction information between global and local tokens. The information contained in the local tokens is integrated into the global token to obtain the visual feature X containing spatial interaction information. gs ;
[0012] S12: Visual feature X gs Spatial pooling is performed, and then a convolutional layer spanning the time dimension is used to generate a temporal importance graph V and update the local tokens, resulting in local tokens containing temporal information.
[0013] S13: Calculate the average value of the local tokens to obtain the global token containing spatial information;
[0014] S14: The global token and local token of each frame are concatenated to generate a new global visual feature X. gst This is then fed into a multilayer perceptron and subjected to residual connections to obtain a coarse-grained token X' containing spatiotemporal information. g ;
[0015] S15: Use a coarse-grained global token containing spatiotemporal information as the final global feature.
[0016] Furthermore, step S2 includes the following steps:
[0017] S21: Adopt The similarity between each local token and the global token is represented as the spatial visual weight of that token;
[0018] S22: Use a sliding window of size a×b to slide across each frame of the image, dividing the local tokens into N groups; for the nth group, there are a×b local tokens in the group, and the set of token indices is represented by U. n Then it corresponds to The subscript set V in n Calculated using a mapping function;
[0019] S23: Calculate the sum of the spatial visual weights of all tokens in each group as the group visual weight.
[0020] S24: Yes Key visual groups were selected using the top-K function.
[0021] S25: Obtaining the key visual group After corresponding to the original image position, a skeleton network is used to extract local block features X. l .
[0022] Furthermore, step S3 includes the following steps:
[0023] S31: In the block fusion module, the mean and standard deviation of the feature map of each block are calculated as style features. A 1D convolution operation is used to aggregate the channel dimensions, and then the Sigmoid function is used to generate a channel importance map G; the local block features X are then... l Multiplying with the channel importance graph G yields the block feature X, which contains key information within the group. lc ;
[0024] S32: Calculate the difference between each feature map and the rest of the feature maps to obtain the difference feature D. Use a residual connection to add inter-group information to the block features, and then use a fully connected hierarchy to generate a discriminative token X that includes interaction information for localization. lcd ;
[0025] S33: Regarding X lcd and the global features obtained in step S1 A cross-attention mechanism is used, followed by a series of convolutional layers to aggregate global and local features. Then, the local features are pooled along the K-dimensional plane to obtain the final local feature X'. l ;
[0026] S34: Transfer local feature X'l With global features Connecting them together yields the video representation X. v .
[0027] Beneficial effects: (1) This invention improves the accuracy of weakly supervised group activity recognition by gradually locating the visual blocks of the entire scene and then further refining the learning of local and global features.
[0028] (2) The present invention designs a new coarse-grained spatial locator that roughly captures the key visual blocks of the entire scene based on the spatial interaction relationship between global and local tokens in nested interactors, thus solving the problem of existing methods using a large number of redundant features.
[0029] (3) This invention designs a new fine-grained spatiotemporal selector that considers the global spatiotemporal context through a cross-attention mechanism and performs fine encoding of local visual block features.
[0030] (4) The present invention can efficiently analyze the category of group activities without relying on fine-grained labels including individual location and individual action category.
[0031] (5) This invention does not use existing detectors to obtain individual positions, and automatically captures and locates key visual parts during training, making it more flexible.
[0032] (6) By first roughly locating the key visual regions and then refining them, the present invention can efficiently and accurately explore the relationships and differences between key visual regions and improve the accuracy of analysis. Attached Figure Description
[0033] Figure 1 This is a flowchart of the method of the present invention;
[0034] Figure 2 This is a structural block diagram corresponding to the method of the present invention;
[0035] Figure 3 The specific steps of the coarse-grained spatial locator in this invention are as follows: (1) represents a sliding window, (2) represents a window sliding step, (3) represents a Top-K selection step, (4) represents a visual block positioning step, and (5) represents a visual block clipping step.
[0036] Figure 4 This is a comparison chart of the experimental results of this invention on the Volleyball dataset with the results of other methods;
[0037] Figure 5 This is a comparison chart of the experimental results of this invention on the NBA dataset with the results of other methods. Detailed Implementation
[0038] The invention will now be further explained with reference to the accompanying drawings.
[0039] like Figures 1 to 3 As shown, the group activity recognition method and model training method provided by the present invention can detect the activity categories of crowds in a video.
[0040] Group activity recognition typically uses individual location labels, then uses RoIAlign to extract individual features, capture interaction information and spatiotemporal variations between individuals, and finally generate a video representation reflecting the interaction and scene information. To reduce reliance on labels and improve the applicability of group activity recognition, weakly supervised group activity recognition methods aim to extract effective key visual features without any additional fine-grained individual supervision information.
[0041] Existing solutions extract visual features through the following typical methods: (a) Person features. This uses off-the-shelf object detectors to generate person location nominations, then extracts person features. However, it heavily relies on the quality of existing detectors and lacks flexibility. (b) Mesh features. This employs a Transformer to learn a set of token embeddings from noisy mesh features with learnable weights, then aggregates them into the overall video representation. However, it encodes all local visual information equally and does not delve into key visual content highly relevant to the activity. In summary, existing methods fail to capture key visual features flexibly and accurately.
[0042] To address the aforementioned problems, this invention provides a detector-free group activity recognition method based on coarse-grained and fine-grained nested learning. This method not only adaptively selects key visual components but also performs deep modeling of the relationships between them. Specifically, the framework first coarsely locates key visual blocks based on their similarity to global information, and then further refines the encoding of these key visual blocks using interaction information and global spatiotemporal evolution information.
[0043] The technical solution of the present invention will be described in detail below with reference to specific embodiments.
[0044] I. A detector-free group activity recognition method based on coarse-grained and fine-grained nested learning
[0045] S1 nested interactor. The T-frame RGB image is cropped to obtain... H and W represent the dimensions of the cropped image, which is then fed into a skeleton network to extract visual features. Where C represents the feature dimension. The visual features are then flattened to obtain... Where N = H' × W'. Each feature can be viewed as a local token, and a learnable global token is defined for each frame. The updated visual features are obtained by concatenating with local tokens. in
[0046] Then for visual feature X g Learnable spatial location encodings are added, and a spatial attention mechanism is employed to explore spatial interaction information between global and local tokens, integrating the information contained in the local tokens into the global token. This process can be represented by the following formula:
[0047]
[0048]
[0049]
[0050] Where e pos This represents a learnable positional embedding. and This represents the parameters of the linear transformation layer in the self-attention operation. It is the attention map for each frame of the image.
[0051] To reduce the number of training parameters, convolution operations are used when capturing spatial dynamics. First, the visual features X containing spatial interaction information are processed. gs Spatial pooling is performed, followed by convolutional layers across the time dimension to generate a temporal importance graph and update local tokens. This process can be represented by the following formula:
[0052]
[0053]
[0054]
[0055] Where V is the temporal importance graph, and Conv represents 1D convolution. The 3 and 1 in the formula represent the kernel size. C represents the output dimension. σ is an activation function. This represents a local token containing timing information. ⊙ represents the element-wise multiplication operation.
[0056] Then, the average value of the local tokens containing temporal information is calculated to obtain the global token containing spatial information:
[0057]
[0058] Finally, the global tokens and local tokens from each frame are concatenated to generate a new global visual feature X. gst This is then fed into a multilayer perceptron (MLP), followed by residual connections to obtain a coarse-grained token X' containing spatiotemporal information.g This process can be represented by the following formula:
[0059] X′ g =X gst +MLP(X gst ).
[0060] in Finally, a coarse-grained global token incorporating spatiotemporal information is used. This is the final global feature.
[0061] S2 Coarse-grained Spatial Localizer. Group activity recognition relies on complex relationships between individuals. To reduce the cost of handling redundant spatial interactions in a scene without bounding box labels, this invention designs a coarse-grained spatial localizer to roughly locate several key visual blocks.
[0062] In the spatial interaction module with nested interactors, the attention matrix A t This represents the similarity between tokens within the same frame. Local tokens with high similarity to global tokens are considered to contain key visual information; therefore, this invention uses the last attention map of spatial interactions obtained in S1 as the spatial importance map. First, let... The similarity between each local token and the global token is used to represent the spatial visual weight of that token. Since a local token may only contain a portion of the key visual information, this invention uses a sliding window of size a×b to divide the local tokens into N groups. For the nth group, it contains a×b local tokens, and the set of token indices can be represented as U. n Using the subscript mapping function Φ(U) n ) = V n Its corresponding similarity vector can be calculated. The subscript set V in n Next, calculate all tokens in each group (index U). n The sum of spatial visual weights is used as the group visual weight. It can be expressed by the following formula:
[0063]
[0064] right The selection of key visual groups using the top-K function can be expressed as:
[0065]
[0066] The topk(·) function selects the first few values.
[0067] After obtaining the positions of these selected groups corresponding to the original image, a skeleton network is used to extract local block features X.l ∈R T×C .
[0068] S3 Fine-grained Spatiotemporal Selector. The fine-grained spatiotemporal selector is used to refine coarse local visual keyblocks obtained from coarse-grained spatial locators, generating efficient local feature representations.
[0069] To fine-tune coarse-grained local features, this invention designs a block fusion module that learns hidden individual features through channel attention. First, the mean and standard deviation of the feature map are calculated for each frame to obtain style features. Then, a 1D convolution operation is applied to the channel dimension, and a sigmoid function is used to obtain the channel importance map G. Local block features X l Multiplying with the channel importance graph G yields the block feature X, which contains key information within the group. lc This process can be represented by the following formula:
[0070] G t =Sigmoid(Conv(cat(μ t ,σ t ))),
[0071] X lc =X l ⊙G,
[0072] Where μ and σ are the mean and derivative, G is the channel importance map, and Conv represents 1D convolution. ⊙ represents element-wise multiplication.
[0073] For each feature map, its difference from the rest of the feature maps is calculated to obtain the difference feature D. A residual connection is then used to add inter-group information to the block features. Finally, a fully connected hierarchy is used to generate a discriminative token X that includes interaction information for localization. lcd This process can be represented by the following formula:
[0074]
[0075] X lcd =FC(X) lc +D).
[0076] Where FC(·) represents the entire chain hierarchy. i={1,2,…,K}, t={1,2,…,T}.
[0077] Next, this invention employs a cross-attention mechanism, which optimizes local features by utilizing global features obtained in nested interactive learners, while preventing information loss.
[0078] For X lcd and the global features obtained in step S1 A cross-attention mechanism is used, followed by a series of convolutional layers to aggregate global and local features. Then, pooling of the local features along the K-dimensional plane yields the final local feature X'. l This process can be represented by the following formula:
[0079]
[0080]
[0081] Finally, the local feature X' l With global features Connecting them together yields the video representation X. v .
[0082] S4 group activity prediction. Representing video as X v It is passed into the classifier for classification.
[0083] II. Model Training Method for Detector-Free Group Activity Recognition Based on Coarse-Grained-Fine-Grained Nested Learning:
[0084] 1. Obtain video clip samples and their corresponding video-level activity tags, which represent the activity categories of a group of people in a video.
[0085] 2. Divide the samples and their labels into two parts proportionally: one part is the training set used to train the model, and the other part is the validation set used to select the model.
[0086] 3. After processing the samples in the training set, the model outputs the prediction results, and the cross-entropy loss is used to calculate the loss between the prediction results and the true labels.
[0087] 4. The model is trained through backpropagation and parameter updates, and inference is tested using a validation set to obtain better model results.
[0088] In the task of identifying group activities, the method of this invention is compared with state-of-the-art methods on two benchmark datasets: the Volleyball dataset and the NBA dataset. For the Volleyball dataset, this invention uses two metrics to evaluate model accuracy: MCA (Multi-Class Accuracy) and Merged MCA (combining the categories "Right Setter" and "Right Pass" into "Right Setter-Pass", and combining "Left Setter" and "Left Pass" into "Left Setter-Pass"). For the NBA dataset, this invention uses two metrics to evaluate model accuracy: MCA (Multi-Class Accuracy) and MPCA (Mean Accuracy per Class).
[0089] For the Volleyball dataset, compared to methods using ResNet-18, this invention achieves a performance improvement of over 1.6% on MCA and comparable results on merged MCA. Compared to methods relying on bounding boxes (using the same detector to obtain nominations), this invention improves MCA by 4.7% without any prior information, demonstrating its powerful learning ability to filter out critical visual patches and extract effective interaction information. Compared to methods without detectors, this method improves MCA by 1.6%, highlighting the importance of local features and the interactions between key visual patches in group activity recognition tasks.
[0090] In the NBA dataset, the scenes are complex, NBA players move faster, and the relationships between them are more intricate. Compared to methods that rely on bounding boxes (using the same detector to obtain nominations), this invention outperforms them by 14.3% on MCA and by 14.3% on MPCA, indicating that the semantic visual patches obtained by this invention are effective and less redundant. Compared to methods that do not use detectors, this method improves MCA by 2.3% and MPCA by 2.1%, demonstrating its remarkable ability to capture spatiotemporal dependencies in complex scenes with fast-moving individuals. These results demonstrate that this invention is robust in capturing key visual patches and handling fast-moving scenes without requiring movement-related labels such as optical flow, trajectories, etc.
[0091] In this method, a segment-based sampling strategy is employed, sampling 5 frames per clip from the Volleyball dataset and 18 frames per clip from the NBA dataset. Each frame is resized to 720×1280. This method uses a ResNet-18 pre-trained on ImageNet as the backbone network, with the feature dimension set to 512. In the interaction nester, this invention uses two spatial and temporal interaction modules to capture spatial and temporal interactions. Specifically, this method uses the last attention map from the spatial interaction as the importance map, which is then fed into a coarse-grained spatial locator to coarsely locate key visual patches. The number of multi-heads in the self-attention layer is set to 4. In the convolutional layers of the temporal interaction, the kernel size is set to 3 and the output dimension to 128 for the first convolutional layer to integrate temporal information while reducing the channel size. Then, the kernel size is set to 1 and the output dimension to 512 for the second convolutional layer to generate the temporal importance map.
[0092] In the coarse-grained spatial locator, our method uses 1×3 overlapping tokens for the NBA dataset and updates the importance map by summing the similarity of the three tokens with the global token in the same frame. For the Volleyball dataset, we use 2×3 overlapping tokens to form groups and update the importance map based on the similarity between the intermediate local token and the global token. The settings differ due to the varying sizes of visual key parts in the videos. Depending on the scene size and the number of key parts, we set the number of key visual groups to 4 for the Volleyball dataset and 6 for the NBA dataset.
[0093] In the fine-grained spatiotemporal selector, a convolutional kernel of size 1 is used in the first convolutional operation to fuse spatial information, and a convolutional kernel of size 3 is used in the second convolutional operation to model cross-channel information. The number of multi-heads in the cross-attention layer is set to 4.
[0094] During training, this method uses the Adam optimizer, setting β1 = 0.9, β2 = 0.999, and ε = 1e-8. The learning rate starts from 1e... -6 to 1e -4 Unequal, weight decay set to 1e -4 Set the batch size for the Volleyball dataset to 6 and the batch size for the NBA dataset to 4.
[0095] As described above, the present invention can be implemented well.
[0096] The above description is only a preferred embodiment of the present invention. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the principle of the present invention, and these improvements and modifications should also be considered within the scope of protection of the present invention.
Claims
1. A method for identifying group activities based on coarse-grained and fine-grained nested learning, characterized in that, Includes the following steps: S1 Nested Interactive Learning: The T-frame RGB images are cropped and then fed into the skeleton network to extract visual features. Each feature is treated as a local token. A learnable global token is defined for each frame and concatenated with the local tokens to update the visual features. The visual features are then fed into the nested interactive network to capture spatiotemporal global information and obtain global features. S2 coarse-grained spatial localization: The spatial visual weight of each local token is represented by the similarity between the local token and the global token. Then, local tokens with high similarity are roughly selected and mapped to the original image, which is to roughly locate the key visual blocks. S3 Fine-grained spatiotemporal selection: The local block features obtained in step S2 are refined using a block fusion module and a cross-attention mechanism; S4 Group Activity Prediction: Input the video representation obtained in step S3 into the classifier for classification.
2. The group activity recognition method based on coarse-grained and fine-grained nested learning according to claim 1, characterized in that, In step S1, a frame of image contains N local tokens, which are concatenated with a global token to obtain new visual features. Where C is the feature dimension; the step of inputting visual features into a nested interactor to capture spatiotemporal global information includes the following steps: S11: Visual feature X g A learnable spatial location encoding is added, and then a spatial attention mechanism is used to explore the spatial interaction information between global and local tokens. The information contained in the local tokens is integrated into the global token to obtain the visual feature X containing spatial interaction information. gs ; S12: Visual feature X gs Spatial pooling is performed, and then a convolutional layer spanning the time dimension is used to generate a temporal importance graph V and update the local tokens, resulting in local tokens containing temporal information. S13: Calculate the average value of the local tokens to obtain the global token containing spatial information; S14: The global token and local token of each frame are concatenated to generate a new global visual feature X. gst This is then fed into a multilayer perceptron and subjected to residual connections to obtain a coarse-grained token X' containing spatiotemporal information. g ; S15: Use a coarse-grained global token containing spatiotemporal information as the final global feature.
3. The group activity recognition method based on coarse-grained and fine-grained nested learning according to claim 1, characterized in that, Step S2 includes the following steps: S21: Adopt The similarity between each local token and the global token is represented as the spatial visual weight of that token; S22: Use a sliding window of size a×b to slide across each frame of the image, dividing the local tokens into N groups; for the nth group, there are a×b local tokens in the group, and the set of token indices is represented by U. n Then it corresponds to The subscript set V in n Calculated using a mapping function; S23: Calculate the sum of the spatial visual weights of all tokens in each group as the group visual weight. S24: Yes Key visual groups were selected using the top-K function. S25: Obtaining the key visual group After corresponding to the original image position, a skeleton network is used to extract local block features X. l .
4. The group activity recognition method based on coarse-grained and fine-grained nested learning according to claim 1, characterized in that, Step S3 includes the following steps: S31: In the block fusion module, the mean and standard deviation of the feature map of each block are calculated as style features. A 1D convolution operation is used to aggregate the channel dimensions, and then the Sigmoid function is used to generate a channel importance map G; the local block features X are then... l Multiplying with the channel importance graph G yields the block feature X, which contains key information within the group. lc ; S32: Calculate the difference between each feature map and the rest of the feature maps to obtain the difference feature D. Use a residual connection to add inter-group information to the block features, and then use a fully connected hierarchy to generate a discriminative token X that includes interaction information for localization. lcd ; S33: Regarding X lcd and the global features obtained in step S1 A cross-attention mechanism is used, followed by a series of convolutional layers to aggregate global and local features. Then, the local features are pooled along the K-dimensional plane to obtain the final local feature X'. l ; S34: Transfer local feature X' l With global features Connecting them together yields the video representation X. v .