Self-supervised group behavior recognition method and system based on global and local comparative learning

Abstract

The invention provides a self-supervised group behavior recognition method and system based on global and local comparative learning, and the method comprises the steps: extracting individual features through local branches, generating a soft mask by employing a multi-head self-attention mask module, separating significant / non-significant individual features through mask pooling, constructing a comparison sample, and carrying out the recognition of a group behavior through the comparison sample; feature alignment is optimized in combination with cosine similarity and local contrast loss; meanwhile, the spatial interaction relation of the behaviorists is captured by utilizing the spatial global Transform of global branches, short-term action and long-term behavior modes are fused through multi-scale time sequence coding, the consistency is optimized by adopting global comparison loss after spatial and temporal characteristics are aggregated, and finally, the two branch characteristics are integrated through global-local comparison loss for comparison, so that the accuracy of the behavior behavior is improved. And the loss weight is automatically adjusted to simplify parameter adjustment. After training is completed, a group behavior recognition result is output through the classifier according to the extracted spatial-temporal features, and the discrimination ability and robustness of the model to complex group behaviors are effectively improved.

Description

Technical Field

[0001] The present invention belongs to the technical field of group behavior recognition, and in particular relates to a self-supervised group behavior recognition method and system based on global-local contrast learning. Background Art

[0002] Crowd behavior recognition is a highly complex task, requiring algorithms to not only understand individual actions in a video but also analyze and understand the interactions between individuals and their relationship with the environment. In complex and dynamic crowd scenes, individual actors may perform different actions that may be independent or interdependent, leading to complex interaction patterns such as cooperation, competition, leadership, and following. To effectively recognize crowd behavior in these scenes, algorithms typically rely on precise annotation of the input video sequences. This means that detailed labeling of individual and group actions is required during the training phase to provide accurate supervision signals. However, methods that rely on individual annotations suffer from several issues: First, they significantly increase the workload and cost of data preparation, as detailed annotations of individuals in each video are required. Second, these methods may not generalize well to new, unseen scenes due to their over-reliance on specific features and behavioral patterns in the training data. Finally, inaccurate or inconsistent individual annotations can directly impact model performance and reliability. Furthermore, methods that rely on individual annotations may face further limitations in practical applications. For example, in real-time applications such as security monitoring or traffic management, there is often insufficient time and resources to perform detailed individual annotations for every video frame.

[0003] In the field of group action recognition, fully supervised learning methods typically rely on detailed individual-level and group-level annotations, including bounding boxes for individuals in videos and identification of interaction pairs. These methods precisely localize and identify individuals in videos, analyze their spatiotemporal characteristics, and integrate these features into group-level representations to achieve action classification. Due to the limitations of fully supervised methods that rely on precise annotations, researchers have initially explored two main approaches to overcome these limitations: 1) Unified learning frameworks, which integrate individual detection and group action recognition into a unified learning framework without relying on bounding boxes. Such frameworks can learn group action representations from video data, reducing the reliance on precise annotations. 2) Weakly supervised learning methods, which do not require detailed individual-level annotations but instead utilize group-level or weak labels for training and inference. Although these methods reduce the need for precise annotations, they still rely on some form of label information, such as group-level labels.

[0004] To reduce reliance on labels for individual actions and group behaviors, researchers have applied self-supervised learning to group behavior recognition. One proposed model, the Contextualized Relation Predictive Model (Con-RPM), which learns spatiotemporal representations of group behavior through self-supervision. This model uses an encoder-decoder architecture to learn spatiotemporal features from videos through predictive coding, enabling effective group behavior recognition with limited annotations. Others have proposed a self-supervised framework based on "Spatial and Temporal Attention Heads" (STAHs) and "Global Spatio-Temporal Contrastive" (GSTCo). This framework independently extracts spatially and temporally aware features and uses a global spatio-temporal contrast loss function to optimize the spatiotemporal feature representation. While self-supervised group behavior recognition has made progress in learning holistic spatiotemporal features, it has not adequately addressed the detailed features of local spatial regions, which are equally important for understanding complex group behaviors.

[0005] Based on the above analysis, the present invention proposes a self-supervised group behavior recognition scheme based on global-local contrastive learning. Summary of the Invention

[0006] In response to the above technical problems, the present invention provides a self-supervised group behavior recognition method and system based on global-local contrastive learning.

[0007] The technical solution adopted by the present invention to solve the technical problem is: A self-supervised group behavior recognition method based on global-local contrastive learning, the method comprising the following steps: S100: Obtain video clips and input them into the backbone network of the feature extraction network. The RoI-Align module of the local branch extracts individual actor features, and the global branch extracts global context features. S200: Input the individual actor features into the multi-head self-attention mask module, first perform feature embedding and generate multiple attention heads through linear transformation, then calculate the feature information of each attention head separately, concatenate the obtained features and perform projection fusion operation to finally generate a soft mask; S300: Separate salient and non-salient individual features in the soft mask through mask pooling, construct positive and negative sample pairs, calculate cosine similarity, and then combine the local contrast loss function to optimize feature space alignment and differentiation; S400: Combines the spatial global Transformer to extract global spatial features, captures temporal coding features of different periods through a multi-scale temporal information encoding module and fuses them to obtain temporal features. Global spatial features and temporal features are aggregated to obtain the final global spatiotemporal features, and a global contrast loss is introduced to optimize model feature consistency. S500: Define the global-local contrast loss to integrate the local and global feature comparisons. When the preset training end conditions are reached, obtain the pre-trained model. Optimize the pre-trained model based on the linear evaluation and fine-tuning evaluation modules to obtain the optimized prediction model. Complete group behavior recognition based on the optimized prediction model, where the prediction model includes a backbone network, a RoI-Align module for obtaining individual features, and a classifier for determining group behavior labels.

[0008] Preferably, the feature extraction network uses Inception-v3 as the backbone network. In S100, the video clip is obtained and input into the backbone network of the feature extraction network. The individual actor features are extracted through the local branch RoI-Align module, including: S110: For each frame in the video sequence, the input video clip is processed by the convolutional network to obtain a convolutional feature map ,in Indicates the length of the video clip, that is, the number of frames it contains. represents the feature dimension, Indicates the height of the image, Indicates the width of the image; S120: In the local branch, in order to extract the features of individual actors from the feature map of each frame, the RoI-Align module is used to crop and align the convolution feature map to extract the features of individual actors. Individual features corresponding to the bounding boxes of each actor; S130: The extracted individual features are then encoded through a linear layer to transform the features from the original dimension Convert to the dimensions required by the model , generating individual actor characteristics .

[0009] Preferably, the global branch in S100 extracts global context features, including: S140: In the global branch, the convolution feature map Transformed into convolutional feature map , and consider the time dimension as the batch size, where Represents the spatial dimension of the feature map. This transformation operation flattens the feature map of each frame into a two-dimensional matrix so that the features of each time step are regarded as an independent spatial feature. S150: Use point-by-point convolution to Channels are mapped to In each semantic group, the label is obtained by the following calculation method: ; in represents the spatial attention matrix, and represents the learnable parameter matrix for linear transformation, Used to The number of channels from Map to semantic groups, The linear transformation used to adjust the feature dimension is The number of channels and individual features The dimensions are consistent, and the above method is used to generate the final global context feature representation , Indicates the number of global tags.

[0010] Preferably, S200 includes: S210: Characterizing Individual Actors Input to the multi-head self-attention mask module, after feature embedding, normalization layer and linear layer, input to the multi-head self-attention block. The multi-head self-attention mechanism calculates the query by the following formula ,key Sum : ; ; ; Among them, for each frame , Indicates that at time step characteristics of individual actors, Converted into multiple attention heads , , , is the learnable parameter matrix of the linear network, with dimension , , Indicates the feature dimension processed by each attention head; S220: Use the softmax function to calculate its attention features: ; in, represents the normalized scale factor; S230: Feature information from different attention head spaces is aggregated into a single feature embedding representation through a concatenation operation. The process is as follows: ; ; in, Represents a splicing operation, Indicates that at time step and attention The parameter transformation matrix on , represents the i-th attention head, Represents the features after being processed by the attention mechanism, Represents the calculated attention feature matrix; S240: After self-attention calculation, the result of attention weighting It is sent to the linear layer for projection adjustment; finally, the Sigmoid function is used to generate a soft mask , and its calculation process is expressed as follows: ; in, represents a linear layer function, Represents a pre-set hyperparameter used to adjust the tolerance to neutral features.

[0011] Preferably, S300 includes: S310: The formula for mask pooling is defined as follows: ; ; ; in, represents the soft mask corresponding to the non-salient features, Indicates the soft mask corresponding to the salient feature, represents the identity matrix, represents the number of individual characteristics, and represents element-wise multiplication, Represents the distinctive characteristics of individual actors, Indicates non-salient characteristics of individual actors; S320: Generate corresponding video frames through mask pooling The significant and non-significant individual feature representations of are regarded as negative sample pairs, and the contrast loss of negative samples in the input video sequence can be expressed as: ; in, , Indicates the frame index number, Indicates the Salient individual features of frames With the Non-salient individual features of the frame The cosine similarity between Compute the contrast loss between the salient feature representations and the non-salient feature representations for all frames in the entire sequence; S330: In the context of local contrastive learning, a positive sample pair is formed by a salient individual feature representation and a salient individual feature representation, or a non-salient individual feature representation and a non-salient individual feature representation, of different frames in the same video sequence. The contrastive loss of the positive sample pair is expressed as: ; ; ; in, represents the contrast loss of the positive sample pair, represents the loss of positive samples in the same salient region, represents the loss of positive samples in the same non-significant region, Indicates the Frame and Cosine similarity of salient features between frames, Indicates the Frame and Cosine similarity of non-salient features between frames; S340: The overall local contrast loss function is expressed as: ; in, Represents the overall local contrast loss.

[0012] Preferably, S400 includes: S410: The Frame time A visual marker is defined as , sent to the spatial global Transformer module, its calculation process is expressed as: ; ; in, Indicates in Global feature input at the frame time, It represents the output features obtained after processing by the multi-head self-attention module and combined with the residual connection. MHSA represents the multi-head self-attention block, LN represents the normalization layer, and MLP consists of a linear mapping layer. Represents the feature vector finally output after passing through the spatial global Transformer module; S420: Input the output of the spatial global Transformer module to the multi-scale temporal information encoding module. The multi-scale temporal information encoding module includes three different branches of temporal convolutional networks. The convolution kernel sizes of the TCNs on different branches are 3, 5, and 7, respectively, to capture short-term, medium-term, and long-term temporal coding features; S430: Fuse the temporal coding features of the three different branches, use 1×1 convolution for channel compression, and splice them to obtain the final fused temporal features , the fusion strategy is expressed as: ; Among them, among them, Indicates three different branches, represents the learnable mapping matrix, Indicates the A temporal convolutional network to capture temporal information at different time scales; S440: Aggregate the output of the spatial global Transformer module with the temporal features output by the multi-scale temporal information encoding module to obtain the final global spatiotemporal feature representation ; S450: Global contrastive learning is used to train the spatial global Transformer and the multi-scale temporal information encoding module. The global contrastive loss function is expressed as: ; in, represents the global contrast loss, , Indicates the index number of the frame, represents the cosine similarity metric function, represents an indicator function, when is 1, otherwise it is 0. For the moment The global spatiotemporal characteristics of represents the video-level global feature representation of the corresponding frame, Indicates time global spatiotemporal characteristics.

[0013] Preferably, the global-local contrast loss is defined in S500 as follows: ; in, and Represents automatic adjustment of relative weights based on task uncertainty, which are used to balance the contributions of different task loss functions during training. and As a regularization term, its purpose is to control the complexity of the model and avoid overfitting.

[0014] A self-supervised group behavior recognition system based on global-local contrastive learning, including a feature extraction module, a multi-head self-attention mask module, a mask pooling module, a spatial global Transformer module, and a group behavior recognition module; The feature extraction module is used to obtain video clips, extract individual actor features through local branches, and extract global context features through global branches; The multi-head self-attention mask module is used to input the individual actor features into the multi-head self-attention mask module. It first performs feature embedding and generates multiple attention heads through linear transformation. Then, the feature information of each attention head is calculated separately. The obtained features are spliced and fused through projection operation to finally generate a soft mask; The mask pooling module is used to separate salient and non-salient individual features in the soft mask, construct positive and negative sample pairs, calculate cosine similarity, and then combine the local contrast loss function to optimize feature space alignment and differentiation; The spatial global Transformer module is used to extract global spatial features. It uses a multi-scale temporal information encoding module to capture temporal coding features of different periods and fuse them to obtain temporal features. It aggregates global spatial features and temporal features to obtain the final global spatiotemporal features. It also introduces a global contrast loss to optimize the consistency of model features. To extract global spatial features, it uses multi-scale temporal convolution to capture different time domain information. After fusing multi-branch features, it aggregates the spatiotemporal representation and introduces a global contrast loss to optimize the consistency of model features. The group behavior recognition module is used to define the global-local contrast loss to integrate the local and global feature comparisons. When the preset training end conditions are reached, a pre-trained model is obtained. The pre-trained model is optimized based on the linear evaluation and fine-tuning evaluation modules to obtain the optimized prediction model. Group behavior recognition is completed based on the optimized prediction model, where the prediction model includes a backbone network, a RoI-Align module for obtaining individual features, and a classifier for determining group behavior labels.

[0015] A computer device includes a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, it implements the steps of a self-supervised group behavior recognition method based on global-local contrastive learning.

[0016] A computer-readable storage medium stores a computer program, which, when executed by a processor, implements the steps of a self-supervised group behavior recognition method based on global-local contrastive learning.

[0017] The above-mentioned self-supervised group behavior recognition method and system based on global-local contrastive learning aims to improve the accuracy and robustness of group behavior recognition by combining local and global contrast perspectives to learn discriminative feature representations. Specifically, a multi-head self-attention mask module is first introduced in the local branch to learn the soft attention mask of individual local features, thereby distinguishing between significant and non-significant local features. Secondly, a spatial global Transformer module is designed in the global branch to capture the spatial interaction relationship between actors in the same frame, thereby providing rich spatial context information for understanding group behavior. In order to capture the temporal dynamics across frames, the present invention adopts a multi-scale temporal information encoding strategy to simultaneously consider the short-term actions and long-term behavior patterns of the actors, so that it can more comprehensively understand the evolution process of group behavior. Finally, a global-local contrast loss function is designed to automatically adjust the weights of each loss function, thereby reducing the workload and complexity of hyperparameter adjustment. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] Figure 1 This is a flow chart of a self-supervised group behavior recognition method based on global-local contrastive learning in one embodiment of the present invention; Figure 2 This is a general framework diagram of a self-supervised group behavior recognition method based on global-local contrastive learning in one embodiment of the present invention; Figure 3 Schematic diagram of a multi-head self-attention mask module according to one embodiment of the present invention; Figure 4 Schematic diagram of a multi-scale temporal information encoding module in one embodiment of the present invention. DETAILED DESCRIPTION

[0019] In order to enable those skilled in the art to better understand the technical solution of the present invention, the present invention is further described in detail below with reference to the accompanying drawings.

[0020] In one embodiment, Figure 1 and Figure 2 As shown, a self-supervised group behavior recognition method based on global-local contrastive learning includes the following steps: S100: Obtain video clips and input them into the backbone network of the feature extraction network. The RoI-Align module of the local branch extracts individual actor features, and the global branch extracts global context features. S200: Input the individual actor features into the multi-head self-attention mask module, first perform feature embedding and generate multiple attention heads through linear transformation, then calculate the feature information of each attention head separately, concatenate the obtained features and perform projection fusion operation to finally generate a soft mask; S300: Separate salient and non-salient individual features in the soft mask through mask pooling, construct positive and negative sample pairs, calculate cosine similarity, and then combine the local contrast loss function to optimize feature space alignment and differentiation; S400: Combines the spatial global Transformer to extract global spatial features, captures temporal coding features of different periods through a multi-scale temporal information encoding module and fuses them to obtain temporal features. Global spatial features and temporal features are aggregated to obtain the final global spatiotemporal features, and a global contrast loss is introduced to optimize model feature consistency. S500: Define the global-local contrast loss to integrate the local and global feature comparisons. When the preset training end conditions are reached, obtain the pre-trained model. Optimize the pre-trained model based on the linear evaluation and fine-tuning evaluation modules to obtain the optimized prediction model. Complete group behavior recognition based on the optimized prediction model, where the prediction model includes a backbone network, a RoI-Align module for obtaining individual features, and a classifier for determining group behavior labels.

[0021] This self-supervised group action recognition method based on global-local contrastive learning first provides a comprehensive overview of both individual actor feature extraction and global context feature extraction. It then further introduces the components of local contrastive learning, primarily including the multi-head self-attention mask module and the local contrastive learning function. It then describes how to construct global contrastive learning to capture global context information. Finally, a global-local contrastive loss function is designed to dynamically adjust the relative weights in the loss function for group action recognition.

[0022] Furthermore, in the field of group behavior recognition, designing an effective feature extraction method is crucial for understanding and analyzing individual interactions and scene contexts in videos. The feature extraction network of the present invention aims to provide rich discriminant information for subsequent comparative learning by extracting local individual behavior features and global context features respectively. Given a video clip ,in Indicates the length of the video clip, that is, the number of frames it contains, 3 represents the typical RGB (red, green, blue) three-channel color model, Indicates the height of the image, that is, the number of pixels in the vertical direction, Indicates the width of the image, that is, the number of pixels in the horizontal direction. In the feature extraction part, the local branch focuses on extracting individual actor features to capture detailed information at the individual level, while the global branch extracts global context features of the entire scene.

[0023] The local branch focuses on extracting features related to individual actors from video frames, providing refined feature representations for group action recognition. These features are crucial for understanding the details of individual actions, and combined with global context features, they can provide a more comprehensive representation of group actions.

[0024] In one embodiment, the feature extraction network uses Inception-v3 as the backbone network. In S100, the video clip is obtained and input into the backbone network of the feature extraction network. The individual actor features are extracted through the RoI-Align module of the local branch, including: S110: For each frame in the video sequence, the input video clip is processed by the convolutional network to obtain a convolutional feature map ,in Indicates the length of the video clip, that is, the number of frames it contains. represents the feature dimension, Indicates the height of the image, Indicates the width of the image; S120: In the local branch, in order to extract the features of individual actors from the feature map of each frame, the RoI-Align module is used to crop and align the convolution feature map to extract the features of individual actors. Individual features corresponding to the bounding boxes of each actor; S130: The extracted individual features are then encoded through a linear layer to transform the features from the original dimension Convert to the dimensions required by the model , generating individual actor characteristics A filter-based tokenizer is used to extract visual tokens, which treats feature channels as a set of signals that are further processed to extract discriminative visual information.

[0025] In the feature extraction process of the global branch, the goal is to capture the global context features of the entire group scene, which are crucial for understanding the overall dynamic and structural information in the scene. In addition to the local detail features of individual actors, the present invention further extracts the global context features from the feature map. Extract global features.

[0026] In one embodiment, extracting global context features through a global branch in S100 includes: S140: In the global branch, the convolution feature map The transformation is the convolution feature map , and consider the time dimension as the batch size. This transformation operation flattens the feature map of each frame into a two-dimensional matrix, where Represents the spatial dimension of the feature map so that the features of each time step are considered as an independent spatial feature; S150: Use point-by-point convolution to Channels are mapped to In each semantic group, the label is obtained by the following calculation method: ; in represents the spatial attention matrix, and represents the learnable parameter matrix for linear transformation, Used to The number of channels from Map to semantic groups, The linear transformation used to adjust the feature dimension is The number of channels and individual features The dimensions are consistent, and the above method is used to generate the final global context feature representation , Indicates the number of global tags.

[0027] Furthermore, in group behavior recognition, although global features can provide the overall pattern of behavior, local features are equally important for capturing key actors or key information in group behavior. For example, in sports games or social activities, subtle individual movements, such as an athlete's gestures or expressions, may have a decisive influence on the recognition of the behavior of the entire scene. However, traditional global feature extraction methods often ignore these subtle but important local changes, resulting in limited ability to recognize and understand behavior in complex scenes. In order to remedy this shortcoming, the present invention proposes a local contrastive learning method to capture rich and discriminative feature representations in group scenes in a self-supervised manner. This part mainly consists of two parts: a multi-head self-attention mask module and a local contrastive learning function.

[0028] The multi-head self-attention mask module utilizes the unique query-key matching function of the multi-head self-attention mechanism to explore the correlation between individual actor features from multiple perspectives, thereby mining the correlation of local features at the feature level. Because the multi-head self-attention mechanism can process features of multiple subspaces in parallel, each "head" can learn a unique feature representation, thereby effectively capturing the complex relationships and interactions between different individual actors. Through the self-attention mechanism, the module is able to identify and emphasize local individual actor features that are critical to understanding specific group behaviors, while suppressing information that is less important for the current group behavior recognition task. In short, by focusing on important local features while ignoring irrelevant features, the model can more accurately distinguish different group behaviors and reduce confusion between behaviors with similar appearance features.

[0029] In one embodiment, Figure 3 As shown, S200 includes: S210: Characterizing Individual Actors Input to the multi-head self-attention mask module, after feature embedding, normalization layer and linear layer, input to the multi-head self-attention block. The multi-head self-attention mechanism calculates the query by the following formula ,key Sum : ; ; ; Among them, for each frame , Indicates that at time step characteristics of individual actors, Converted into multiple attention heads . , , is the learnable parameter matrix of the linear network, with dimension , , Indicates the feature dimension processed by each attention head; S220: Use the softmax function to calculate its attention features: ; in, represents the normalized scale factor; S230: Feature information from different attention head spaces is aggregated into a single feature embedding representation through a concatenation operation. The process is as follows: ; ; in, Represents a splicing operation, Indicates that at time step and attention The parameter transformation matrix on , represents the i-th attention head, Represents the features after being processed by the attention mechanism, Represents the calculated attention feature matrix; S240: After self-attention calculation, the result of attention weighting It is fed into a linear layer for projection adjustment; S250: Finally, a soft mask is generated using the Sigmoid function , and its calculation process is expressed as follows: ; in, represents a linear layer function, Represents a pre-set hyperparameter used to adjust the tolerance to neutral features.

[0030] Furthermore, the core goal of local contrastive learning is to enhance the model's ability to identify the features of individual actors within a video frame through a contrastive learning strategy. This module uses a soft mask generated by a multi-head self-attention mask module to distinguish and extract salient individual features from non-salient individual features in feature embedding. The soft mask setting not only reflects the relative importance of an individual in the group, but also reveals its key actions in the behavior pattern. Then, mask pooling technology is further used to process the input individual features. , respectively, to obtain the salient features of individual actors and non-salient features .

[0031] In one embodiment, S300 includes: S310: The formula for mask pooling is defined as follows: ; ; ; in, represents the soft mask corresponding to the non-salient features, represents the soft mask corresponding to the salient feature, represents the identity matrix, represents the number of individual characteristics, and represents element-wise multiplication, Represents the distinctive characteristics of individual actors, Indicates non-salient characteristics of individual actors; S320: Generate corresponding video frames through mask pooling The significant and non-significant individual feature representations of are regarded as negative sample pairs, and the contrast loss of negative samples in the input video sequence can be expressed as: ; in, , Indicates the index number of the frame, Indicates the Salient individual features of frames With the Non-salient individual features of the frame The cosine similarity between Compute the contrast loss between the salient feature representations and the non-salient feature representations for all frames in the entire sequence; In video sequences, since consecutive frames typically exhibit temporal and motion coherence, this video coherence can be exploited to design a contrastive learning strategy for positive sample pairs. In the context of local contrastive learning, positive sample pairs are formed by combining salient individual feature representations with salient individual feature representations, or non-salient individual feature representations with non-salient individual feature representations, from different frames in the same video sequence. This strategy helps the model learn the similarities and differences between temporally consecutive video frames, thereby better understanding the individual behavior patterns in the video. S330: In the context of local contrastive learning, a positive sample pair is formed by a salient individual feature representation and a salient individual feature representation, or a non-salient individual feature representation and a non-salient individual feature representation, of different frames in the same video sequence. The contrastive loss of the positive sample pair is expressed as: ; ; ; in, represents the contrast loss of the positive sample pair, represents the loss of positive samples in the same salient region, represents the loss of positive samples in the same non-significant region, Indicates the Frame and Cosine similarity of salient features between frames, Indicates the Frame and Cosine similarity of non-salient features between frames; S340: The overall local contrast loss function is expressed as: ; in, Represents the overall local contrast loss.

[0032] Furthermore, in the field of group action recognition, the extraction and analysis of global features is crucial for understanding the complex dynamics of a scene. Global features encompass not only the macroscopic layout of the entire scene but also the relationships and interaction patterns between individual elements within it. This paper follows the model framework of previously fully supervised group action recognition and employs a spatial global transformer (SGT-Former) to model spatial relationships, aiming to extract and integrate global contextual information from video frames.

[0033] In one embodiment, Figure 4 As shown, S400 includes: S410: The Frame time A visual marker is defined as , sent to the spatial global Transformer module, its calculation process is expressed as: ; ; in, Indicates in Global feature input at the frame time, It represents the output features obtained after processing by the multi-head self-attention module and combined with the residual connection. MHSA represents the multi-head self-attention block, LN represents the normalization layer, and MLP consists of a linear mapping layer. Represents the feature vector finally output after passing through the spatial global Transformer module; in this way, the SGT-Former module can effectively process and integrate global context information from different visual markers, thereby improving the expressiveness of features and the generalization ability of the model; In the task of group activity recognition based on video data, temporal information is crucial for understanding the global dynamics of behavior in complex scenes. Because group behavior evolves over time, the contribution and importance of different time points to the overall group activity recognition vary. This paper further proposes a Multi-scale Temporal Encoding Module (MTEM) to effectively integrate contextual information from different time points.

[0034] S420: The output of the spatial global Transformer module is fed into a multi-scale temporal information encoding module. The multi-scale temporal information encoding module consists of three different branches of temporal convolutional networks (TCNs). The convolution kernel sizes of the TCNs in different branches are 3, 5, and 7, respectively, capturing short-term, medium-term, and long-term temporal information. This provides rich temporal context for understanding the overall dynamics of group behavior. S430: Fuse the temporal coding features of the three different branches, use 1×1 convolution for channel compression, and splice them to obtain the final fused temporal features , the fusion strategy is expressed as: ; Among them, among them, Indicates three different branches, represents the learnable mapping matrix, Indicates the A temporal convolutional network to capture temporal information at different time scales; S440: Aggregate the output of SGT-Former with the temporal features of MTEM output to obtain the final global spatiotemporal feature representation ; After capturing the global spatiotemporal features, Global Contrastive Learning (GCL) is used to train the proposed spatial global Transformer and multi-scale temporal information encoding module. In a specific video frame, the global feature representation (frame level) of the current frame should be consistent with the global feature representation (video level) of the consecutive frames in the same video sequence, and should be distinguishable from the global feature representation (frame level) of the video frames at other moments. Global spatiotemporal characteristics of For example, its frame level representation Its corresponding video-level representation attract each other, the latter is represented by pooling frames Obtained, its different video frame level representation and Mutually exclusive.

[0035] S450: Global contrastive learning is used to train the spatial global Transformer and the multi-scale temporal information encoding module. The global contrastive loss function is expressed as: ; in, represents the global contrast loss, , Indicates the index number of the frame, represents the cosine similarity metric function, represents an indicator function, when is 1, otherwise it is 0. For the moment The global spatiotemporal characteristics of represents the video-level global feature representation of the corresponding frame, Indicates time global spatiotemporal characteristics.

[0036] In this way, the global contrastive learning function not only promotes the model's understanding of temporal coherence in video sequences, but also further enhances the ability to recognize behavioral changes at different time points, thereby improving the recognition accuracy of group behavior recognition.

[0037] Furthermore, the global-local contrast loss function integrates local contrast loss with global contrast loss, while taking into account local behavior details and global context in group scenes, thereby promoting the robustness and discrimination of multi-view feature representation in group behavior recognition. This joint optimization strategy reduces the workload and complexity of model tuning by automatically adjusting the weights of the loss function, reducing the manual adjustment of hyperparameters. Therefore, the present invention dynamically adjusts the relative weights of each loss function by constructing a multi-task learning framework. The multi-task loss function adjusts the relative weights of each task loss function by maximizing the Gaussian likelihood, while taking into account the uncertainty of task dependence, so that the model can automatically adjust the weights according to the difficulty and uncertainty of each task.

[0038] In one embodiment, the global-local contrast loss is defined in S500 as follows: ; in, and Represents automatic adjustment of relative weights based on task uncertainty, which are used to balance the contributions of different task loss functions during training. and As a regularization term, its purpose is to control model complexity and avoid overfitting. All network parameters and task uncertainty weights are trainable and optimized using the gradient backpropagation algorithm.

[0039] To evaluate the effectiveness of the proposed self-supervised learning framework, after self-supervised pre-training, the pre-trained model was subjected to linear and fine-tuning evaluations. Performance analysis and ablation experiments were conducted on three widely used datasets: VD, CAD, and WVD-M. Multi-class accuracy (MCA) and mean per-class accuracy (MPCA) were used as evaluation metrics. This paper first describes the implementation details of self-supervised group action recognition, including model inputs, hyperparameters, and model training parameters. Next, the proposed method is compared with fully supervised methods and several weakly supervised methods. A series of ablation experiments were then conducted on a volleyball dataset. Finally, a visualization analysis of the proposed model is presented.

[0040] In self-supervised learning, the quality of the learned representation cannot be directly evaluated due to the lack of direct supervisory signals to measure the accuracy of the model. To solve this problem, researchers usually adopt an indirect evaluation strategy: first, the model is pre-trained using self-supervised learning; then, the obtained model is applied to a specific supervised learning task and fine-tuned on this basis to evaluate the effectiveness of the self-supervised learning feature representation. The present invention designs two evaluation methods to evaluate the performance of group behavior recognition. The first method is linear evaluation, in which the network structure is frozen, the network weights learned in the pre-training phase are retained, and no layer of the network is updated. Only the linear classification layer is trained to facilitate the direct use of the model initialized by self-supervised learning. The second method is fine-tuning evaluation, which fine-tunes the entire network structure and uses limited labeled data to optimize the model parameters, thereby further improving the performance of the model on specific tasks.

[0041] After self-supervised pre-training, the inference components in the local and global branches are removed, and the model quality is evaluated through linear evaluation and fine-tuning of the pre-trained model. The prediction model architecture consists of a pre-trained backbone network, a RoI-Align module for extracting individual features, and a classifier for determining group behavior labels. The goal of the entire training process is to minimize the cross-entropy loss function, thereby reducing the prediction error of the behavior category.

[0042] Model input: This invention uses the Inception-v3 model pre-trained on the ImageNet dataset as the backbone network. During model training, we do not focus on locating the corresponding individual actors in space, but follow methods such as DIN (Dynamic Inference Network), ARG (Actor Relationship Graph), and MLST-Former (Multi-level Spatiotemporal Transformer Model) to obtain the bounding boxes of individual actors. For fairness in comparison, the resolution of all video frames on the three datasets was uniformly scaled to Each video sequence samples consecutive Frames are used as input to the model, where The value is set to 10. The dimensions used are The linear embedding layer obtains the individual actor features, and the RoIAlign (cropping size is 5×5) technology is used to crop and align them to extract the feature map of the individual actor.

[0043] Hyperparameters: For the Collective Activity Dataset (CAD), the maximum number of actors in the scene For the volleyball dataset VD and the weakly supervised volleyball dataset WVD-M, the maximum number of actors in the scene The number of global tags in global context feature extraction , the multi-head self-attention mask module is equipped with The soft mask generation network has For the spatial global Transformer module, we stack three layers of spatial global Transformer blocks, equip each layer with four attention heads, and set the dropout rate to 0.1.

[0044] Model training: All model training on the datasets uses stochastic gradient descent combined with the Adam algorithm to learn network parameters, and the hyperparameters are fixed at , ,and . In the linear evaluation and fine-tuning evaluation, dropout = 0.9 is applied on the last layer of the classifier. The initial learning rate of the Adam optimizer is set to 0.001, and the learning rate will decay to 0.0001 when the loss on the validation set no longer decreases significantly, that is, when it reaches a stable state. In the testing phase, the present invention follows the testing protocol of the standard baseline model and extracts sequences of the same length as in the training phase from the video for evaluation. The performance comparison with existing advanced algorithms on the collective activity dataset is shown in Table 1. / indicates that the results are not provided, bold indicates the best performance, and underline indicates the algorithm ranked second on this indicator.

[0045] Table 1

[0046] In this paper, we conducted performance analysis experiments comparing the proposed method with existing state-of-the-art methods on three crowd action recognition datasets (CAD, VD, and WVD-M). We also compared the performance of a detectorless model on the NBA dataset. Since there are very few results for self-supervised crowd action recognition, we compared the performance of our method with fully supervised methods and some weakly supervised methods to verify its effectiveness.

[0047] 1) Comparison of Experimental Results on the Collective Activity Dataset (CAD): Table 1 shows the experimental results of our proposed method on the collective activity dataset. These results demonstrate that our proposed method achieves the best recognition performance on a non-fully supervised learning model, even surpassing some fully supervised learning methods on the MCA and MPCA metrics. Specifically, using fine-tuning evaluation, our proposed method achieves an accuracy only 1.2% lower than the fully supervised learning method ASTFormer on the MCA metric. On the MPCA metric, our proposed method outperforms the GSTCo method (using the same backbone network) by 4.6% and 0.7% on the linear and fine-tuning evaluations, respectively. This experimental result demonstrates that global-local contrastive learning can provide discriminative feature representations for the model and confirms the effectiveness of local contrastive learning for group activity recognition. Table 2 shows a performance comparison with existing state-of-the-art algorithms on the volleyball dataset and the weakly supervised volleyball combined dataset. "M-MCA" represents the result of merging the "passing" and "setting" categories in WVD-M. / indicates that results are not provided. Bold indicates the best performance, and underlined indicates the second-best performing algorithm on that metric.

[0048] Table 2

[0049] 2) Comparison of VD Experimental Results on the Volleyball Dataset: Table 2 shows the experimental results of our method on the volleyball dataset. These results demonstrate that the proposed self-supervised model outperforms other weakly self-supervised methods in recognition performance, as measured by both linear and fine-tuning evaluations. While the GSTCo method uses a global spatiotemporal contrastive loss to aggregate spatial and temporal features for group action recognition, our method improves performance by combining local and global contrastive learning, demonstrating the importance of local contrastive learning in group action recognition. This performance comparison demonstrates that local contrastive learning focuses on capturing and learning the characteristics of individual actors within a video frame, while global contrastive learning focuses on the global contextual features of the entire video scene. Through this multi-perspective combination, our method achieves a more comprehensive understanding of group action in videos, resulting in superior performance in group action recognition. Furthermore, our model achieves comparable recognition accuracy to the state-of-the-art self-supervised GAR model on the volleyball dataset, but still lags behind the fully supervised GAR model. This result suggests that while self-supervised learning has achieved some performance, there is still room for improvement.

[0050] 3) Comparison of experimental results on the weakly supervised volleyball combined dataset WVD-M: The present invention compares the performance of the self-supervised method of the present invention with other group activity recognition methods with different supervision methods on the weakly supervised volleyball combined dataset. The experimental results reveal two main phenomena: (1) In WVD-M, the recognition results after merging the two action categories of "first pass" and "second pass setting" are improved compared with the performance on the volleyball dataset. This result shows that by merging action categories with similar characteristics, the commonalities in group behavior can be more effectively captured, thereby improving the recognition accuracy; (2) The self-supervised training results of the present invention method on WVD-M are better than those of other weakly supervised method models. The performance of the linear evaluation (95.3%) is 0.1% higher than that of the LRMM method, and even better than most of the fully supervised methods in Table 2. This result shows the effectiveness of self-supervised learning in dealing with limited labeled data.

[0051] The present invention proposes a self-supervised group behavior recognition method based on global-local contrastive learning, which aims to learn the representation of group behavior through the SSL scheme. In the feature extraction network, the local branch focuses on extracting the features of local individual actors, and the global branch focuses on extracting the global context features of the entire scene. In the pre-training stage, discriminative feature representations are learned by combining global contrastive learning and local contrastive learning to improve the accuracy and robustness of group behavior recognition. A large number of experimental results show that the network framework proposed in the present invention exhibits better performance than the existing SOTA method.

[0052] The above-mentioned self-supervised group behavior recognition method based on global-local contrastive learning has the following technical effects: 1) We propose a novel contrastive learning framework for self-supervised group behavior recognition based on global-local contrastive learning, which can simultaneously focus on the local details of individual behaviors and the global patterns of group behaviors to learn discriminative feature representations.

[0053] 2) Design a multi-head self-attention mask module to learn salient and non-salient individual features in local branches. By generating soft attention masks, this module can focus more on key behavioral features that are important for group behavior recognition while ignoring less important information.

[0054] 3) A global-local contrast loss function is constructed to automatically adjust the weights of various loss functions. The effectiveness of the proposed method is verified by comparing its recognition performance with existing weakly supervised and self-supervised methods on multiple mainstream group behavior recognition datasets.

[0055] In one embodiment, a self-supervised group behavior recognition system based on global-local contrastive learning is also provided, comprising a feature extraction module, a multi-head self-attention mask module, a mask pooling module, a spatial global Transformer module, and a group behavior recognition module; The feature extraction module is used to obtain video clips, extract individual actor features through local branches, and extract global context features through global branches; The multi-head self-attention mask module receives individual actor features, embeds the features, and generates multiple attention heads through linear transformation. It then calculates the query-key association weights, normalizes them through softmax, concatenates the features, and projects and fuses them. Finally, a sigmoid function is used to generate a soft mask. The mask pooling module is used to separate salient and non-salient individual features in the soft mask, construct positive and negative sample pairs, calculate cosine similarity, and then combine the local contrast loss function to optimize feature space alignment and differentiation; The spatial global Transformer module is used to extract global spatial features. It captures information from different time domains through multi-scale temporal convolution, aggregates spatiotemporal representations after fusing multi-branch features, and introduces a global contrast loss to optimize model feature consistency. The group behavior recognition module is used to define the global-local contrast loss to integrate the local and global feature comparisons. When the preset training end conditions are reached, a pre-trained model is obtained. The pre-trained model is optimized based on the linear evaluation and fine-tuning evaluation modules to obtain the optimized prediction model. Group behavior recognition is completed based on the optimized prediction model, where the prediction model includes a backbone network, a RoI-Align module for obtaining individual features, and a classifier for determining group behavior labels.

[0056] Regarding the specific definition of a self-supervised group behavior recognition system based on global-local contrastive learning, please refer to the definition of a self-supervised group behavior recognition method based on global-local contrastive learning above, which will not be repeated here. The various modules in the above-mentioned self-supervised group behavior recognition system based on global-local contrastive learning can be implemented in whole or in part by software, hardware and a combination thereof. The above-mentioned modules can be embedded in or independent of the processor in the computer device in the form of hardware, or can be stored in the memory of the computer device in the form of software, so that the processor can call and execute the operations corresponding to the above modules.

[0057] A computer device includes a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, it implements the steps of a self-supervised group behavior recognition method based on global-local contrastive learning.

[0058] A computer-readable storage medium stores a computer program, which, when executed by a processor, implements the steps of a self-supervised group behavior recognition method based on global-local contrastive learning.

[0059] Those skilled in the art will appreciate that all or part of the processes in the above-described method embodiments can be implemented by instructing the relevant hardware through a computer program. The computer program can be stored in a non-volatile computer-readable storage medium. When executed, the computer program can include the processes of the above-described method embodiments. Any reference to memory, storage, database, or other media used in the embodiments provided herein may include at least one of non-volatile and volatile memory. Non-volatile memory may include read-only memory (ROM), magnetic tape, floppy disk, flash memory, or optical storage. Volatile memory may include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM).

[0060] The above is a detailed introduction to the self-supervised group behavior recognition method and system based on global-local contrastive learning provided by the present invention. This article uses specific examples to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only used to help understand the core idea of the present invention. It should be pointed out that for ordinary technicians in this technical field, without departing from the principles of the present invention, the present invention can also be improved and modified, and these improvements and modifications also fall within the scope of protection of the claims of the present invention.

Claims

1. A self-supervised group behavior recognition method based on global-local contrastive learning, characterized by: The method comprises the following steps: S100: Obtain video clips and input them into the backbone network of the feature extraction network. The RoI-Align module of the local branch extracts individual actor features, and the global branch extracts global context features. S200: Input the individual actor features into the multi-head self-attention mask module, first perform feature embedding and generate multiple attention heads through linear transformation, then calculate the feature information of each attention head separately, concatenate the obtained features and perform projection fusion operation to finally generate a soft mask; S300: Separate salient and non-salient individual features in the soft mask through mask pooling, construct positive and negative sample pairs, calculate cosine similarity, and then combine the local contrast loss function to optimize feature space alignment and differentiation; S400: Combines the spatial global Transformer to extract global spatial features, captures temporal coding features of different periods through a multi-scale temporal information encoding module and fuses them to obtain temporal features. Global spatial features and temporal features are aggregated to obtain the final global spatiotemporal features, and a global contrast loss is introduced to optimize model feature consistency. S500: Define the global-local contrast loss to integrate the local and global feature comparisons. When the preset training end conditions are reached, obtain the pre-trained model. Optimize the pre-trained model based on the linear evaluation and fine-tuning evaluation modules to obtain the optimized prediction model. Complete group behavior recognition based on the optimized prediction model, where the prediction model includes a backbone network, a RoI-Align module for obtaining individual features, and a classifier for determining group behavior labels.

2. The self-supervised group behavior recognition method based on global-local contrastive learning according to claim 1 is characterized in that: The feature extraction network uses Inception-v3 as the backbone network. S100 obtains video clips and inputs them into the backbone network of the feature extraction network. The RoI-Align module of the local branch extracts the features of individual actors, including: S110: For each frame in the video sequence, the input video clip is processed by the convolutional network to obtain a convolutional feature map ,in Indicates the length of the video clip, that is, the number of frames it contains. represents the feature dimension, Indicates the height of the image, Indicates the width of the image; S120: In the local branch, in order to extract the features of individual actors from the feature map of each frame, the RoI-Align module is used to crop and align the convolution feature map to extract the features of individual actors. Individual features corresponding to the bounding boxes of each actor; S130: The extracted individual features are then encoded through a linear layer to transform the features from the original dimension Convert to the dimensions required by the model , generating individual actor characteristics .

3. The self-supervised group behavior recognition method based on global-local contrastive learning according to claim 2 is characterized in that: The global branch in S100 extracts global context features, including: S140: In the global branch, the convolution feature map Transformed into convolutional feature map , and consider the time dimension as the batch size, where Represents the spatial dimension of the feature map. This transformation operation flattens the feature map of each frame into a two-dimensional matrix so that the features of each time step are regarded as an independent spatial feature. S150: Use point-by-point convolution to Channels are mapped to In each semantic group, the label is obtained by the following calculation method: ； in represents the spatial attention matrix, and represents the learnable parameter matrix for linear transformation, Used to The number of channels from Map to semantic groups, The linear transformation used to adjust the feature dimension is The number of channels and individual features The dimensions are consistent, and the above method is used to generate the final global context feature representation , Indicates the number of global tags.

4. The self-supervised group behavior recognition method based on global-local contrastive learning according to claim 3 is characterized in that: S200 includes: S210: Characterizing Individual Actors Input to the multi-head self-attention mask module, after feature embedding, normalization layer and linear layer, input to the multi-head self-attention block. The multi-head self-attention mechanism calculates the query by the following formula ,key Sum : ； ； ； Among them, for each frame , Indicates that at time step characteristics of individual actors, Converted into multiple attention heads , , , is the learnable parameter matrix of the linear network, with dimension , , Indicates the feature dimension processed by each attention head; S220: Use the softmax function to calculate its attention features: ； in, represents the normalized scale factor; S230: Feature information from different attention head spaces is aggregated into a single feature embedding representation through a concatenation operation. The process is as follows: ； ； in, Represents a splicing operation, Indicates that at time step and attention The parameter transformation matrix on , represents the i-th attention head, Represents the features after being processed by the attention mechanism, Represents the calculated attention feature matrix; S240: After self-attention calculation, the result of attention weighting It is sent to the linear layer for projection adjustment; finally, the Sigmoid function is used to generate a soft mask , and its calculation process is expressed as follows: ； in, represents a linear layer function, Represents a pre-set hyperparameter used to adjust the tolerance to neutral features.

5. The self-supervised group behavior recognition method based on global-local contrastive learning according to claim 4 is characterized in that: S300 includes: S310: The formula for mask pooling is defined as follows: ； ； ； in, represents the soft mask corresponding to the non-salient features, Indicates the soft mask corresponding to the salient feature, represents the identity matrix, represents the number of individual characteristics, and represents element-wise multiplication, Represents the distinctive characteristics of individual actors, Indicates non-salient features of individual actors; S320: Generate corresponding video frames through mask pooling The significant and non-significant individual feature representations of are regarded as negative sample pairs, and the contrast loss of negative samples in the input video sequence can be expressed as: ； in, , Indicates the frame index number, Indicates the Salient individual features of frames With the Non-salient individual features of the frame The cosine similarity between Compute the contrast loss between the salient feature representations and the non-salient feature representations for all frames in the entire sequence; S330: In the context of local contrastive learning, a positive sample pair is formed by a salient individual feature representation and a salient individual feature representation, or a non-salient individual feature representation and a non-salient individual feature representation, of different frames in the same video sequence. The contrastive loss of the positive sample pair is expressed as: ; ； ； in, represents the contrast loss of the positive sample pair, represents the loss of positive samples in the same salient region, represents the loss of positive samples in the same non-significant region, represents the cosine similarity of the salient features between the i-th frame and the j-th frame, represents the cosine similarity of non-salient features between the i-th frame and the j-th frame; S340: The overall local contrast loss function is expressed as: ; in, Represents the overall local contrast loss.

6. The self-supervised group behavior recognition method based on global-local contrastive learning according to claim 5 is characterized in that: S400 includes: S410: The Frame time A visual marker is defined as , sent to the spatial global Transformer module, its calculation process is expressed as: ； ； in, Indicates in Global feature input at the frame time, It represents the output features obtained after processing by the multi-head self-attention module and combined with the residual connection. MHSA represents the multi-head self-attention block, LN represents the normalization layer, and MLP consists of a linear mapping layer. Represents the feature vector finally output after passing through the spatial global Transformer module; S420: Input the output of the spatial global Transformer module to the multi-scale temporal information encoding module. The multi-scale temporal information encoding module includes three different branches of temporal convolutional networks. The convolution kernel sizes of the TCNs on different branches are 3, 5, and 7, respectively, to capture short-term, medium-term, and long-term temporal coding features; S430: Fuse the temporal coding features of the three different branches, use 1×1 convolution for channel compression, and splice them to obtain the final fused temporal features , the fusion strategy is expressed as: ； Among them, among them, Indicates three different branches, represents the learnable mapping matrix, Indicates the A temporal convolutional network to capture temporal information at different time scales; S440: Aggregate the output of the spatial global Transformer module with the temporal features output by the multi-scale temporal information encoding module to obtain the final global spatiotemporal feature representation ; S450: Global contrastive learning is used to train the spatial global Transformer and the multi-scale temporal information encoding module. The global contrastive loss function is expressed as: ； in, represents the global contrast loss, , Indicates the index number of the frame, represents the cosine similarity metric function, represents an indicator function, when is 1, otherwise it is 0. For the moment The global spatiotemporal characteristics of represents the video-level global feature representation of the corresponding frame, Indicates time global spatiotemporal characteristics.

7. The self-supervised group behavior recognition method based on global-local contrastive learning according to claim 6 is characterized in that: Global-local contrast loss is defined in S500 as follows: ； in, and Represents automatic adjustment of relative weights based on task uncertainty, which are used to balance the contributions of different task loss functions during training. and As a regularization term, its purpose is to control the complexity of the model and avoid overfitting.

8. A self-supervised group behavior recognition system based on global-local contrastive learning, characterized by: It includes feature extraction module, multi-head self-attention mask module, mask pooling module, spatial global Transformer module and group behavior recognition module; The feature extraction module is used to obtain video clips and input them into the backbone network of the feature extraction network. The RoI-Align module of the local branch extracts individual actor features, and the global branch extracts global context features; The multi-head self-attention mask module is used to input the individual actor features into the multi-head self-attention mask module. It first performs feature embedding and generates multiple attention heads through linear transformation. Then, the feature information of each attention head is calculated separately. The obtained features are spliced and fused through projection operation to finally generate a soft mask; The mask pooling module is used to separate salient and non-salient individual features in the soft mask, construct positive and negative sample pairs, calculate cosine similarity, and then combine the local contrast loss function to optimize feature space alignment and differentiation; The spatial global Transformer module is used to extract global spatial features. It uses a multi-scale temporal information encoding module to capture temporal coding features of different periods and fuse them to obtain temporal features. It aggregates global spatial features and temporal features to obtain the final global spatiotemporal features. It also introduces a global contrast loss to optimize the consistency of model features. To extract global spatial features, it uses multi-scale temporal convolution to capture different time domain information. After fusing multi-branch features, it aggregates the spatiotemporal representation and introduces a global contrast loss to optimize the consistency of model features. The group behavior recognition module is used to define the global-local contrast loss to integrate the local and global feature comparisons. When the preset training end conditions are reached, a pre-trained model is obtained. The pre-trained model is optimized based on the linear evaluation and fine-tuning evaluation modules to obtain the optimized prediction model. Group behavior recognition is completed based on the optimized prediction model, where the prediction model includes a backbone network, a RoI-Align module for obtaining individual features, and a classifier for determining group behavior labels.

9. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.

Images