Group behavior identification method based on cross-modal space-time self-supervision

Abstract

The invention discloses a group behavior identification method based on cross-modal space-time self-supervision, which relates to the technical field of computer vision, and comprises the following steps: preprocessing video data, obtaining an RGB modal individual representation matrix containing space-time reasoning by using an RGB modal space-time interaction network, obtaining a skeleton modal individual representation matrix by using a skeleton modal space-time interaction network, and obtaining a group behavior identification result by using the skeleton modal individual representation matrix. Prediction of an identification result is output through a linear transformation function; a self-supervised learning method is adopted, and a contrast learning task of cross-modal space-time interaction characteristics is designed, so that the dependence on labeled data is reduced, and the model can autonomously learn a space-time interaction mode in a group behavior task; rGB image data and skeleton data are combined, fusion processing is carried out through a space-time interaction network, and the influence of view angle changes on the recognition result is reduced so as to improve the recognition precision.

Description

Technical Field

[0001] The present invention relates to the field of computer vision technology, and in particular to a group behavior recognition method based on cross-modal spatiotemporal self-supervision. Background Art

[0002] Group action recognition (GAR) is the problem of analyzing and understanding the interactive behaviors of multiple individuals in a multi-person scene video sequence. Unlike the human behavior recognition task, it requires accurate analysis of the possible mutual influence, interaction, and interdependence between individuals. GAR based on computer vision involves cutting-edge research technologies in multiple fields such as pattern recognition, representation learning, and video understanding. It not only has far-reaching significance in theoretical research, but also shows a wide range of application value in practical application scenarios such as intelligent video retrieval, sports event analysis, and intelligent security monitoring. In addition, GAR has played a positive role in promoting public safety in areas such as traffic safety management and criminal investigation.

[0003] Before 2016, the research on group behavior recognition mainly focused on the traditional manual feature modeling method, and did not achieve significant results. In recent years, due to the progress of sensors and cameras and the development of deep learning technology, research results in this field have made significant progress. In the context of deep learning technology, according to the development of the field and the use of sample labels by the method, it can be divided into group behavior recognition methods based on fully supervised deep learning and group behavior recognition methods based on self-supervised deep learning strategies.

[0004] Among them, fully supervised learning has made some progress in group behavior recognition, but this method is largely limited by the need for a large amount of accurately labeled data. The complexity of group behavior lies in the fact that it involves many participants, and each individual and their interactions in each frame of the video need to be located and labeled. The labeling cost is much higher than that of a single-person behavior dataset; and the existing self-supervised deep learning strategies mostly use a single data modality to model individual behaviors and individual interactions, ignoring the complexity and diversity of individual and group behaviors. In the process of video data collection, there will be changes in perspective, mutual occlusion between group members, and interference from irrelevant people in complex environments, which increases the difficulty of group behavior analysis.

[0005] Based on this, the present invention aims to provide a group behavior recognition method based on cross-modal spatiotemporal self-supervision to solve two problems: how to use skeleton data to enhance the representation of group behavior patterns and how to reduce the dependence on group behavior annotations. Summary of the invention

[0006] In order to make up for the shortcomings of the prior art problems, the purpose of the present invention is to provide a group behavior recognition method based on cross-modal spatiotemporal self-supervision, which adopts a self-supervised learning method and reduces the dependence on labeled data by designing a comparative learning task of cross-modal spatiotemporal interaction features, so that the model can autonomously learn the spatiotemporal interaction patterns in group behavior tasks; RGB image data and skeleton data are combined and fused through a spatiotemporal interaction network to improve recognition accuracy.

[0007] To achieve the above object, the present invention provides the following technical solutions:

[0008] A group behavior recognition method based on cross-modal spatiotemporal self-supervision includes the following steps:

[0009] (1) Obtaining multiple long videos of group behaviors and preprocessing the videos;

[0010] (2) Construct an interactive network based on cross-modal spatiotemporal self-supervision, which consists of an RGB modality spatiotemporal interaction network and a skeleton modality spatiotemporal interaction network. The RGB modality spatiotemporal interaction network extracts multi-scale feature maps of each frame through the Inception-v3 network, and then infers the RGB modality individual representation through the spatiotemporal interaction network. The skeleton modality spatiotemporal interaction network uses HRNet to extract the skeleton information of the individual in each frame, and then obtains the skeleton modality individual representation through the spatiotemporal interaction network.

[0011] (3) Construct a self-supervised multimodal spatiotemporal contrast loss for training a multimodal spatiotemporal self-supervised interaction network. The extracted features are attracted and repelled by the self-supervised multimodal spatiotemporal contrast loss to train the Inception-v3 network.

[0012] (4) After preprocessing, the real group scene video is input into the trained Inception-v3 network, and the recognition result is output through a linear transformation function.

[0013] The pretreatment in step (1) of the present invention comprises:

[0014] Divide a long video with multiple group behaviors into a series of video clips containing a single group behavior, each sequence clip is represented as a tensor X∈R T×3×H×W , use the social scene understanding network SSU to detect and obtain the individual bounding box of each person in each frame image in the video clip.

[0015] In the present invention, the RGB modality spatiotemporal interaction network includes an Inception-v3 network, a region of interest alignment RoIAlign, and a spatiotemporal Transformer model, and a self-attention mechanism is used to model the spatiotemporal evolution of the interaction relationship in group behavior;

[0016] The skeleton modality spatiotemporal interaction network includes a high-resolution posture estimation network HRNet, a spatiotemporal graph convolution model, and a spatiotemporal Transformer model. The spatiotemporal graph convolution model extracts the spatiotemporal characteristics of individual behaviors under the skeleton modality, reduces the impact of perspective changes on recognition results, and inputs the spatiotemporal Transformer to infer spatiotemporal correlation information.

[0017] The data processing process of the RGB modality spatiotemporal interaction network in step (2) of the present invention is as follows:

[0018] 1-1) Inception-v3 network is used as the backbone to extract the multi-scale feature map of each frame, and the RoIAlign method is used to process the individual bounding box in the multi-scale feature map of each frame to obtain the individual features, and the individual features are embedded into the D-dimensional space. After stacking the individual features, the color image individual representation matrix x is obtained. c ∈R T×N×D ;

[0019] 1-2) Represent the color image individual matrix x c ∈R T×N×D Transformed to and

[0020] 1-3) The features of N individuals in the tth frame Input into the spatial Transformer network, assign importance weights to the relationships between individuals and incorporate them into the characteristics of each individual to reflect the influence of each individual in the group interaction behavior and obtain the individual behavior representation containing spatial interaction information Simplified three-module process, see the following formula:

[0021]

[0022] Among them, SPE is spatial position encoding, which is used to supplement the individual position relationship in space; MHSA is a multi-head self-attention mechanism, which is used to infer the spatial interaction of individuals in the scene; FFN is a feedforward network, which is used to further learn the individual interaction in high-dimensional space; LN is a module and performs layer regularization operation;

[0023] 1-4) The characteristics of the nth individual at T moments Input into the temporal Transformer, and go through the same operation as the three modules in step 2) to obtain the individual behavior representation with cross-frame temporal evolution

[0024] 1-5) and Transform and fuse to obtain the RGB modality individual representation matrix containing spatiotemporal reasoning

[0025] The data processing process of the skeleton modality spatiotemporal interaction network in step (2) of the present invention is specifically as follows:

[0026] 2-1): Introduce the high-resolution posture estimation network HRNet to extract the skeleton information of the individual in each frame image, construct the skeleton individual representation matrix corresponding to the original image, and convert the skeleton individual representation matrix x s ∈R T×N×K×3 Transformed to and

[0027] 2-2): Represent the matrix x for N individuals in the tth frame s ∈R T×N×K×3 Constructing the spatial relationship graph G of human joints t =(V t ,E t ), V t = {v ti |v ti ,i=1,…K joint} and the edge set E t ={e ij |i,j=1,…K joint};v ti Contains node coordinates and confidence, stacked to form e ij Represents the joint node v ti and v tj The relationship between t ; Obtained through spatial graph convolution inference

[0028]

[0029] 2-3): Use Construct time node graph G n =(V n ,E n ), where V n = {v ti |v ti ,i=1,…T} and the edge set E n ={e tt′ |t,t′=1,…T,t′≠t}; as in step 2), the spatial graph convolution process is stacked to form And construct the relationship matrix A n ;

[0030]

[0031] 2-4): and Transform and fuse to obtain the individual representation matrix containing spatiotemporal skeleton reasoning

[0032] 2-5): Referring to steps 1-2) to 1-5), the individual representation matrix represented by the spatiotemporal graph convolution Input into the spatiotemporal Transformer for further spatiotemporal relationship reasoning to obtain the skeleton modality individual representation matrix

[0033] In the present invention, the self-supervised multimodal spatiotemporal contrast loss is constructed as follows:

[0034] (1) The individual representation matrix after multimodal spatiotemporal interactive reasoning and Fusion is performed in spatial and temporal dimensions, eliminating dimensions N and T to obtain RGB modal spatiotemporal interaction representations. and Skeleton modality spatiotemporal interactive representation and

[0035] (2) Spatial interaction characteristics of the two modes and Apply spatial contrast loss to explore spatial dependencies, consider image modality interaction features and skeleton modality interaction pairs in the same frame as positive samples, and image space interaction features and skeleton space interaction pairs in different frames as negative samples;

[0036] Taking the tth frame of the image sequence in spatial interaction contrast as an example, the group interaction representation in the tth frame of the skeleton modality is brought closer to increase the similarity, while the individual interaction representations of other frames in the image sequence are excluded to reduce the similarity; therefore, the spatial interaction contrast loss formula is as follows

[0037]

[0038] in is the exponent of cosine similarity; 1 [i≠t] ∈{0,1} is an indicator function that is assigned a value of 1 if and only if i≠n; and They are the individual spatial interaction features extracted in the same frame in the image modality and the skeleton modality;

[0039] (3) Correspondingly, for the temporal interaction characteristics of the two modes and Use temporal contrast loss to explore individual temporal consistency, consider the RGB modality temporal feature and skeleton modality temporal feature pairs of the same individual as positive samples, and consider the RGB modality temporal feature and skeleton modality temporal feature pairs of different individuals as negative samples;

[0040] Taking the nth individual in the temporal correlation comparison as an example, it is necessary to increase the similarity by pulling in the temporal representation of the nth individual in the skeleton modality, while rejecting the temporal representation of other individuals in the image sequence to reduce the similarity; therefore, the temporal correlation comparison loss is formulated as

[0041]

[0042] in and They represent the individual temporal features extracted from the image data and skeleton data of the same individual. In the time dimension, the change of individual features over time also has a significant impact on the evolution of group behavior;

[0043] (4) The final MSTCo loss is composed of the spatial interaction contrast loss and the temporal contrast correlation loss, and the formula is expressed as:

[0044]

[0045] Among them, λ is a hyperparameter used to adjust the weights.

[0046] Compared with the prior art, the present invention has the following beneficial effects:

[0047] 1. The present invention combines RGB image data and skeleton data, and performs fusion processing through a spatiotemporal interactive network, thereby reducing the impact of viewing angle changes on recognition results and improving recognition accuracy.

[0048] 2. The present invention adopts a self-supervised learning method and reduces the dependence on labeled data by designing a comparative learning task of cross-modal spatiotemporal interaction features, so that the model can autonomously learn the spatiotemporal interaction patterns in group behavior tasks.

[0049] 3. The present invention includes an interactive network of RGB modality and skeleton modality, uses spatiotemporal Transformer and graph convolutional network to perform fine-grained representation of individuals, and uses contrastive learning strategy to model the interaction relationship between individuals.

[0050] 4. The MSTCo loss function designed in the present invention promotes the interactive consistency of the model learning group behavior in different modal spatiotemporal scenarios and enhances the generalization ability of the model. BRIEF DESCRIPTION OF THE DRAWINGS

[0051] Figure 1 It is a schematic diagram of the working method of the present invention.

[0052] Figure 2 This is the confusion matrix diagram of the present invention on Volleyball and Collective. DETAILED DESCRIPTION

[0053] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0054] The technical solutions in the embodiments of the present invention will be described clearly and completely below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than all the embodiments.

[0055] A group behavior recognition method based on cross-modal spatiotemporal self-supervision, specifically including:

[0056] 1. Identification Method Framework

[0057] The multimodal spatiotemporal self-supervised interaction network framework STMI includes a dual-path network structure consisting of an RGB modality spatiotemporal interaction network (CSTIN) and a skeleton modality spatiotemporal interaction network (SSTIN), which is used to explore the interaction patterns between different modal group behaviors. The method framework is as follows Figure 1 As shown in the figure. First, a joint spatiotemporal Transformer is used to form an RGB modal interaction network, and a self-attention mechanism is used to model the spatiotemporal evolution of interaction relationships in group behavior. Second, a spatiotemporal GCN network is constructed to extract the spatiotemporal characteristics of individual behavior under the skeleton modality, reduce the impact of perspective changes on the recognition results, and input into the spatiotemporal Transformer for inference of spatiotemporal correlation information. Then, a self-supervised multimodal spatiotemporal contrast loss (MSTCo) is constructed using the individual spatiotemporal representation after inference based on the RGB and skeleton interaction networks to learn the consistent expression of individual time dimension and spatial interaction dimension in group behavior. Finally, the model is trained using the spatiotemporal contrast loss constructed in the two modal networks.

[0058] (II) Pretreatment process

[0059] Preprocessing requires dividing a long video with multiple group behaviors into a series of video clips containing a single group behavior, and detecting and obtaining the individual bounding box of each person in each frame image in the video clip.

[0060] Step 1): Continuous image frames are used as sample inputs of the model method to obtain richer dynamic information. Long videos with multiple group behaviors are divided into a series of video sequence segments containing a single group behavior, so as to analyze the group behavior pattern more directly. Each sequence segment can be represented as a tensor X∈R T×3×H×W, where each frame image in the sequence contains 3 channels, T is the number of frames in the sequence, and H×W are the height and width of each frame image respectively.

[0061] Step 2): In order to obtain the precise location information and representation of group behavior in sequence frames, the social scene understanding network SSU detection method is used to obtain the bounding box of the characters in each frame.

[0062] 3. Multimodal interactive reasoning

[0063] Group activities are complex and diverse, usually involving complex interactions of multiple individuals in time and space. The changes in information such as interactions and movement relationships between these individuals in the evolution of time and space are important forms of defining different group behaviors. In the spatial dimension, potential clues such as individual positions and postures may have a significant impact on group behavior. For example, in a volleyball match, the setter and the main attacker can constitute the overall offensive behavior state; in the temporal dimension, the temporal context is also crucial for the identification of group behavior. For example, in a volleyball match, the distinction between individual feints and offensive behaviors needs to be judged in the temporal context. Therefore, in order to better understand the dynamic pattern of group behavior in the spatiotemporal dimension, a spatiotemporal interaction inference network STMI is constructed to model the interaction relationship between individuals, including a spatiotemporal interaction network CSTIN based on the RGB modality and a spatiotemporal interaction network SSTIN based on the skeleton modality. Data of different modalities are complementary in the recognition of behavior patterns. CSTIN is designed to extract individual spatiotemporal interaction information of the RGB modality, and SSTIN is designed to extract individual spatiotemporal interaction information of the skeleton modality.

[0064] X∈R T×3×H×W As input to CSTIN and SSTIN, the spatiotemporal interactions between these individuals in the sequence are learned in different modality data.

[0065] 1. Image modality interactive reasoning

[0066] Transformer has a powerful self-attention mechanism that can effectively capture and focus on the relationship between individuals and perform well even in the absence of prior knowledge. STMI uses the spatiotemporal Transformer model to deeply explore the spatiotemporal relationship of groups in RGB modal sample sequences. The spatial Transformer focuses on learning the interaction relationship between individuals in a single frame, and the temporal Transformer is used to explore the temporal context information across frames.

[0067] Step 1): Inception-v3 is used as the backbone network to extract the multi-scale feature map of each frame image. and In order to align with GAR's previous work [5,7,9] Make a comparison.

[0068] X1,X2←Inception-v3(X) (1)

[0069] Step 2): Use the RoIAlign method to process the frame feature map to obtain the individual features in each frame individual frame, and embed the individual features into the D-dimensional space. After stacking the individual features, the color image individual representation matrix x is obtained. c ∈R T×N×D , N corresponds to the maximum number of individuals detected in the frame. In order to maintain the consistency and integrity of the matrix, when the number of individuals detected in a certain frame is less than N, zero vector embedding is used to fill the remaining individual features.

[0070] x c ←RoIAlign(X1,X2)(2)

[0071] Step 3): Considering that simply integrating the spatiotemporal information into a whole for processing may weaken the network's ability to process high-dimensional spatiotemporal features. c ∈R T×N×D Transformed to and Input into the spatial and temporal Transformer respectively.

[0072] Step 4): In order to infer the spatial interaction relationship of individuals, the features of N individuals in the tth frame are Input into the spatial Transformer network, assign importance weights to the relationships between individuals and incorporate them into the characteristics of each individual to reflect the influence of each individual in the group interaction behavior and obtain the individual behavior representation containing spatial interaction information The process can be simplified to three modules, see formulas (3) to (5).

[0073]

[0074] Among them, the spatial position encoding (SPE) is used to supplement the individual position relationship in space (reference), the multi-head self-attention mechanism (MHSA) is used to infer the spatial interaction of individuals in the scene, the feedforward network (FFN) is used to further learn the individual interaction in high-dimensional space, and the modules and both perform layer regularization (LN) operation.

[0075] Step 5): In order to infer the long-term correlation of individuals, the features of the nth individual at T moments are Input to the temporal Transformer network, similar to step 2), after three processes, the individual behavior representation with cross-frame temporal evolution is obtained.

[0076] step6): and Transform and fuse to obtain the RGB modality individual representation matrix containing spatiotemporal reasoning

[0077] 2. Skeleton modal interactive reasoning

[0078] Accurate identification of individual behavior plays a key role in determining group behavior. The internal relationships in human skeleton data can be used to obtain more fine-grained action information and express human behavior more accurately. However, in the study of group activity recognition, skeleton-based human behavior information has not received sufficient attention. In order to capture the details of individual behavior, STMI uses a spatiotemporal graph convolution model to deeply mine the spatiotemporal information of individuals in the skeleton modal sample sequence. This information is inferred to form a skeleton modal individual representation, and a spatiotemporal Transformer model is used to infer the spatiotemporal relationship of the group in the skeleton modal sample sequence.

[0079] Step 1): Different from the previous contrast learning method that uses data enhancement methods such as grayscale transformation and image rotation, a high-resolution posture estimation network (HRNet) that can generate high-quality skeleton images is introduced to extract skeleton information from the individual frames in each frame of the image, and a skeleton individual representation matrix x corresponding to the original image is constructed. s ∈R T×N×K×3 , where the number of individual skeleton joints is K=18.

[0080] x s ←HRNet(X)(6)

[0081] Through this preprocessing method, a unified representation is constructed for each group behavior sequence fragment, which can retain the temporal and spatial information between individuals to a certain extent, facilitating further understanding and analysis of group behavior.

[0082] Step 2): In order to learn more fine-grained human spatiotemporal features, the skeleton individual representation matrix x s ∈R T×N×K×3 Transformed to and Input into spatial and temporal graph convolutions respectively.

[0083] Step 3): Accurately obtaining the spatial relationship between human joints can significantly improve the accuracy of individual behavior recognition. Constructing the spatial relationship graph G of human joints t =(V t ,E t ), V t = {v ti|v ti ,i=1,…K joint} and the edge set E t ={e ij |i,j=1,…K joint}. ti Contains node coordinates and confidence, stacked to form e ij Represents the joint node v ti and v tj The relationship between t . After spatial graph convolution inference, we get

[0084]

[0085] Step 4): In order to capture the temporal dynamics of a joint node in the nth individual at T moments, use Construct time node graph G n =(V n ,E n ), where V n = {v ti |v ti ,i=1,…T} and the edge set E n ={e tt′ |t,t′=1,…T,t′≠t}. Similar to the spatial graph convolution process, stacking forms And construct the relationship matrix A n .

[0086]

[0087] step5): and Transform and fuse to obtain an individual representation matrix containing spatiotemporal skeleton reasoning

[0088] Step 6): Referring to formulas (3) to (5), the individual representation matrix represented by the spatiotemporal graph convolution Input into the spatiotemporal Transformer for further spatiotemporal relationship reasoning to obtain the skeleton modality individual representation matrix

[0089] 4. Spatiotemporal Self-Supervised Contrastive Learning

[0090] In order to improve the accuracy of modeling relationships between individuals in the absence of prior knowledge and make full use of the fine-grained information contained in multimodal data, a teacher-student network structure is constructed to promote the cooperation of image modal interaction and skeleton modal interaction, and a self-supervised multimodal spatiotemporal contrastive loss (MSTCo) training network is designed to promote the model to learn the interaction consistency of group behavior in different modal spatiotemporal scenarios. Self-supervised contrastive learning methods are widely used in visual tasks such as video processing and action recognition. This method encourages the model to learn representations that can capture the essential characteristics of video content without relying on task-specific annotation information. The resulting feature representations often have good generalization capabilities and can be more effectively transferred to different group behavior recognition tasks. The individual representation matrix that has undergone multimodal spatiotemporal interaction reasoning is converted into a matrix. and Perform dimension fusion to construct RGB modal spatiotemporal interaction representations and Skeleton modality spatiotemporal interactive representation and Loss calculation using MSTCo:

[0091] Step 1): For the spatial interaction features of the two modes and Apply spatial contrast loss to explore spatial dependencies, consider image modality interaction features and skeleton modality interaction pairs in the same frame as positive samples, and image space interaction features and skeleton space interaction pairs in different frames as negative samples;

[0092] Taking the tth frame of the image sequence in spatial interaction contrast as an example, we need to pull the group interaction representation in the tth frame of the skeleton modality closer to increase the similarity, while rejecting the individual interaction representations of other frames in the image sequence to reduce the similarity. Therefore, the spatial interaction contrast loss formula is as follows

[0093]

[0094] in is the exponent of the cosine similarity. 1 [i≠t] ∈{0,1} is an indicator function that evaluates to 1 if and only if i≠n. and They are the individual spatial interaction features extracted in the image modality and skeleton modality in the same frame respectively.

[0095] In the spatial dimension, the interaction characteristics between individuals show obvious differences with different behavior types. Taking volleyball games as an example, when performing a batting behavior, the interaction between batters and blockers is stronger. There is a potential antagonistic relationship between these two types of athletes. This spatial interaction relationship between individuals will change in different group behaviors. Based on this observation, a formula is proposed to explore the correlation between individual spatial interactions between different data modalities.

[0096] Step 2): Correspondingly, for the temporal interaction features of the two modes and We use temporal contrast loss to explore individual temporal consistency, considering pairs of RGB modality temporal features and skeleton modality temporal features of the same individual as positive samples, and pairs of RGB modality temporal features and skeleton modality temporal features of different individuals as negative samples.

[0097] Taking the nth individual in the temporal correlation comparison as an example, it is necessary to increase the similarity by pulling in the temporal representation of the nth individual in the skeleton modality, while rejecting the temporal representation of other individuals in the image sequence to reduce the similarity. Therefore, the temporal correlation comparison loss is formulated as

[0098]

[0099] in and Represent the individual temporal features extracted from the image data and skeleton data of the same individual. In the time dimension, the change of individual features over time also has a significant impact on the evolution of group behavior. Taking a volleyball game as an example, when a player hits the ball, if an offensive player changes from standing to jumping in time sequence, while the defensive player remains in a standing moving state and does not jump, it constitutes a feint; if an offensive player changes from standing to jumping in time sequence, while the defensive player performs a jumping block from a standing state, it constitutes an offensive behavior. Therefore, individual temporal related representations can also be used as a discriminative representation reflecting the time dimension. The formula can explore the dependence of group behavior on individual temporal representations by maximizing the temporal similarity between the same individual in image data and skeleton data while comparing the temporal representation pairs of different individuals.

[0100] The network can be trained in an end-to-end manner to predict the behavior of each individual and the group behavior. The final MSTCo loss is composed of spatial interaction contrast loss and temporal contrast correlation loss, and the formula is expressed as:

[0101]

[0102] where λ is a hyperparameter used to adjust the weights, and we simply set λ = 1 in all experiments. After pre-training, the backbone network is able to extract unique spatiotemporal features that can be transferred to various downstream tasks. Finally, the performance of our self-supervised learning method on group activity recognition is evaluated by training a nonlinear transformer on the backbone network with frozen parameters.

[0103] (V) Verification experiment

[0104] The overall multi-class classification accuracy (MCA) is first used to evaluate the model performance. In order to more realistically evaluate the effectiveness of the present invention in verifying each group behavior category on the Volleyball and Collective datasets, the mean precision per class (MPCA) evaluation method is added to alleviate the evaluation problem caused by the imbalance of the number of class samples to a certain extent, and formally expressed as formula,.

[0105] MCA=T total / N total (12)

[0106]

[0107] (5.1) Experimental Dataset

[0108] The present invention adopts two widely used group behavior recognition benchmark datasets, Volleyball and Collective, to test, evaluate and analyze the algorithm to verify its effectiveness.

[0109] 1. Volleyball Dataset (VD) [3]

[0110] Volleyball is based on a collection of public YouTube volleyball game videos, including 4830 video clips from 55 volleyball games, of which 3493 clips are used for training and 1337 are used for validation and testing. Each individual in each frame is assigned an action type, such as Waiting, Setting, Digging, Failing, Spiking, Blocking, Jumping, Standing, Moving; and also includes the group behavior type of each group, such as right-pass, right-spike, right-set, right-winpoint, left-pass, left-spike, left-set, left-winpoint.

[0111] 2. Collective Behavior Dataset (CAD) [1]

[0112] The Collective dataset is collected by low-resolution handheld cameras in street and indoor scenes, and contains 44 video segments with a total of 2481 clips. It contains 6 types of individual action labels, such as NA, Crossing, Queuing, Walking, Talking, and Waiting; and 5 types of group behavior labels, such as Crossing, Queuing, Walking, Talking, and Waiting. During the experiment, according to [6,7,12], the group behavior categories Crossing and Walking were merged into Moving. Two-thirds of the samples in the dataset were used as training sets, and the rest were used as validation sets and test sets.

[0113] (5.2) Experimental environment and parameter settings

[0114] The experiments were conducted on Ubuntu 22.04 system, PyTorch deep learning framework and NVIDIA GeForce RTX3080 GPU.

[0115] Firstly, the image size in the Volleyball dataset is adjusted to 720×1280, and the image size in the Collective dataset is adjusted to 480×720. Five frames are selected in each video clip, including two frames before and after the middle frame, and the number of individuals in each frame is set to 12.

[0116] Secondly, in the processing of RGB modality samples, the preprocessed Inception_v3 network is used as the model backbone network, and the RoIAlign technology is used to adjust the feature map size of the backbone output to 5×5, and then each individual is encoded as a 1024-dimensional feature.

[0117] In addition, in the skeleton modality samples, the HRNet model pre-trained based on the COCO dataset is used to extract each individual skeleton, and the spatiotemporal graph convolution is used to represent the individual skeleton as a 1024-dimensional feature.

[0118] Finally, the number of spatiotemporal graph convolution layers in the network is set to 10, and the number of spatiotemporal Transformer layers is set to 1. In the training of the overall model, the Adam learner is used for the Volleyball dataset, and the training is performed at 2×10 -4 The network is trained with a learning rate of 2×10 for the Collective dataset. The Adam learner is also used for the Collective dataset, but the learning rate is 2×10 -4 The learning rate is 5×10 in 90 to 150 epochs. -4 The learning rate is 1×10 from 151 to 200 epochs. -5The learning rate of the network is

[0119] (5.3) Experimental results analysis

[0120] In order to verify the effectiveness of the present invention, sample training data from the Volleyball dataset and the Collective dataset were used for pre-training in a self-supervised comparison manner, and then fine-tuning training was performed only on the backbone network to evaluate the group behavior recognition ability of the model.

[0121] Table 2 Ablation experiment of the present invention on Volleyball (%)

[0122]

[0123] In the first experiment, various ablation experiments were conducted on the method to verify the effectiveness of the method based on spatiotemporal self-supervised contrastive learning and training methods for the backbone network. First, a model baseline was set to learn the data set in a fully supervised manner; second, the spatiotemporal interactive reasoning module in the combined method was added to obtain the network structure with the strongest representation ability; finally, the designed cross-modal spatiotemporal contrastive loss was used to perform unlabeled learning on the backbone network to verify the effectiveness of the self-supervised learning method. The measurement results are shown in Table 2.

[0124] The model baseline is composed of Inception_v3 pre-trained on ImageNet as the backbone, combined with RoIAlign and classifier; ST-T(a) means that based on the pre-trained Inception_v3, the temporal Transformer is used in the RGB modality for interactive relationship reasoning, and the classifier is applied for performance evaluation; ST-T(b) means that the spatial Transformer is used for interactive relationship reasoning; ST-T(c) means that the space-time Transformer is used for interactive relationship reasoning; ST-G means that based on the model baseline, the space-time graph convolution is used for interactive relationship reasoning. The results show that the proposed method can be used to represent the individual representation of the skeleton modality; ST-GI represents the use of spatiotemporal Transformer to reason about the interaction relationship of individuals in the skeleton modality based on ST-G; DPST represents the combination of ST-T(c) and ST-G to fuse the individual interaction relationship reasoning of the RGB modality and the skeleton modality; CSTIN(a) represents that on the basis of DPST, the backbone network is self-supervised for training the training set through MSTCo, and only the training classifier is fine-tuned; CSTIN(b) represents the fine-tuning of the backbone network projection head based on DPST; CSTIN(c) represents the fine-tuning of the backbone network based on DPST.

[0125] Table 3 Performance comparison on different data ratios

[0126]

[0127] In the second experiment, we evaluated different data ratios on the Volleyball dataset. The network was trained using randomly sampled data of different ratios. As shown in Table 3, the network still has certain recognition performance when the training data is reduced, indicating that the constructed self-supervised contrastive learning method has good generalization ability in extracting universal spatiotemporal representations of group behaviors.

[0128] Table 4 Comparison with mainstream methods on the Volleyball dataset (%)

[0129]

[0130] In the third experiment, the performance of the proposed method was compared with previous work on the Volleyball and Collective datasets to verify the effectiveness of the model. Table 4 lists the results of the comparison with the most advanced models on the Volleyball dataset using the MCA and MPCA evaluation methods. It can be seen that the performance of the proposed method is better than most mainstream methods based on RGB data modality for learning. Although there is a certain performance difference compared with the latest methods that use RGB data, optical flow data, and skeleton data at the same time, the recognition performance of the proposed method is not much different in the methods that also use RGB data and skeleton data, and has a certain degree of comparability.

[0131] Table 5 lists the results of comparison with the most advanced models on the Collective dataset using the MCA and MPCA evaluation methods. It can be seen that the performance of the present invention is within the mainstream model performance range compared with many mainstream methods based on RGB data modality, based on RGB data, optical flow data and skeleton data, and based on RGB data and skeleton data.

[0132] The backbone network pre-trained by the cross-modal spatiotemporal self-supervised comparison method has good recognition performance after a certain degree of fine-tuning with the cross entropy loss, which greatly alleviates the high computational cost and real-time loss caused by the use of complex networks.

[0133] Table 5 Comparison with mainstream methods on the Collective dataset (%)

[0134]

[0135] In the fourth experiment, the confusion matrix of the present invention on the Volleyball and Collective datasets is visualized. Figure 2 It can be seen that based on has good distinguishability.

[0136] [1]CHOI W,SHAHID K,SAVARESE S.What are they doing?:Collectiveactivity classification using spatio-temporal relationship among people[C].2009 IEEE 12th International Conference on Computer Vision Workshops,ICCVWorkshops,2009:1282-1289.

[0137] [2]ZHAI X,HU Z,YANG D,et al.Spatial Temporal Network for Image andSkeleton Based Group Activity Recognition[C].Proceedings of the AsianConference on Computer Vision,2022:20-38.

[0138] [3]IBRAHIM M S,MURALIDHARAN S,DENG Z,et al.A Hierarchical DeepTemporal Model for Group Activity Recognition[C].2016 IEEE Conference onComputer Vision and Pattern Recognition(CVPR),2016:1971-1980.

[0139] [4]SHU T,TODOROVIC S,ZHU S-C.Cern:confidence-energy recurrent networkfor group activity recognition[C],2017:5523-5531.

[0140] [5]WU J,WANG L,WANG L,et al.Learning Actor Relation Graphs for GroupActivity Recognition[C].2019 IEEE / CVF Conference on Computer Vision andPattern Recognition(CVPR),2019:9956-9966.

[0141] [6]YAN R,XIE L,TANG J,et al.HiGCIN:Hierarchical Graph-based CrossInference Network for Group Activity Recognition[[J].IEEE Transactions onPattern Analysis and Machine Intelligence,2020,45(6):6955-6968.

[0142] [7]YUAN H,NI D,WANG M.Spatio-Temporal Dynamic Inference Network forGroup Activity Recognition[J],2021,:.

[0143] [8]GAVRILYUK K,SANFORD R,JAVAN M,et al.Actor-Transformers for GroupActivity Recognition[C].Proceedings of the IEEE / CVF Conference on ComputerVision and Pattern Recognition,2020:839-848.

[0144] [9]LI S,CAO Q,LIU L,et al.GroupFormer:Group Activity Recognition withClustered Spatial-Temporal Transformer[C].2021 IEEE / CVF InternationalConference on Computer Vision(ICCV),2021:13648-13657.

[0145]

[10] HAN M,ZHANG D J,WANG Y,et al.Dual-AI:Dual-path Actor InteractionLearning for Group Activity Recognition[C].2022IEEE / CVF Conference onComputer Vision and Pattern Recognition(CVPR),2022:2980-2989.

[0146]

[11] DU Z,WANG X,WANG Q.Self-supervised global spatio-temporalinteraction pre-training for group activity recognition[J].IEEE Transactionson Circuits and Systems for Video Technology,2023.

[0147]

[12] YAN R,TANG J,SHU X,et al.Participation-Contributed TemporalDynamic Model for Group Activity Recognition[C].MM'18:ACM MultimediaConference,2018:1292-1300.

[0148]

[13] QI M,WANG Y,QIN J,et al.stagNet:An Attentive Semantic RNN forGroup Activity and Individual Action Recognition[J].IEEE Transactions onCircuits and Systems for Video Technology,2020,30(2):549-565.

[0149]

[14] AZAR S M,ATIGH M G,NICKABADI A,et al.Convolutional relationalmachine for group activity recognition[C],2019:7892-7901.

[0150]

[15] ZHENG S, JAYASUMANA S, ROMERA-PAREDES B, et al. Conditional randomfields as recurrent neural networks [C]. Proceedings of the IEEE international conference on computer vision, 2015: 1529-1537.

[0151] What is described above is only a preferred specific implementation manner of the present invention, but the protection scope of the present invention is not limited thereto. Any technician familiar with the technical field can make equivalent replacements or changes according to the technical scheme and inventive concept of the present invention within the technical scope disclosed by the present invention, which should be covered by the protection scope of the present invention.

Claims

1. A group behavior recognition method based on cross-modal spatiotemporal self-supervision, characterized in that: The following steps are involved: (1) Obtaining multiple long videos of group behaviors and preprocessing the videos; (2) Construct an interactive network based on cross-modal spatiotemporal self-supervision, which consists of an RGB modality spatiotemporal interaction network and a skeleton modality spatiotemporal interaction network. The RGB modality spatiotemporal interaction network extracts multi-scale feature maps of each frame image through the Inception-v3 network, and then infers the RGB modality individual representation through the spatiotemporal interaction network. The skeleton modality spatiotemporal interaction network uses HRNet to extract the skeleton information of the individual in each frame, and then obtains the skeleton modality individual representation through the spatiotemporal interaction network; (3) Construct a self-supervised multimodal spatiotemporal contrast loss for training a multimodal spatiotemporal self-supervised interaction network. The extracted features are attracted and repelled by the self-supervised multimodal spatiotemporal contrast loss to train the Inception-v3 network. (4) After preprocessing, the real group scene video is input into the trained Inception-v3 network, and the recognition result is output through a linear transformation function.

2. The group behavior recognition method based on cross-modal spatiotemporal self-supervision according to claim 1 is characterized in that: The preprocessing in step (1) includes: Divide a long video with multiple group behaviors into a series of video clips containing a single group behavior, and each sequence clip is represented as a tensor X∈R T×3×H×W , use the social scene understanding network SSU to detect and obtain the individual bounding box of each person in each frame image in the video clip.

3. The group behavior recognition method based on cross-modal spatiotemporal self-supervision according to claim 1 is characterized in that: The RGB modality spatiotemporal interaction network includes an Inception-v3 network, a region of interest alignment RoIAlign, and a spatiotemporal Transformer model, and uses a self-attention mechanism to model the spatiotemporal evolution of interaction relationships in group behavior; The skeleton modality spatiotemporal interaction network includes a high-resolution posture estimation network HRNet, a spatiotemporal graph convolution model, and a spatiotemporal Transformer model. The spatiotemporal graph convolution model extracts the spatiotemporal characteristics of individual behaviors under the skeleton modality, reduces the impact of perspective changes on recognition results, and inputs the spatiotemporal Transformer to infer spatiotemporal correlation information.

4. The method for group behavior recognition based on cross-modal spatiotemporal self-supervision according to claim 3 is characterized in that: The data processing process of the RGB modality spatiotemporal interaction network in step (2) is as follows: 1-1) Inception-v3 network is used as the backbone to extract the multi-scale feature map of each frame, and the RoIAlign method is used to process the individual bounding box in the multi-scale feature map of each frame to obtain the individual features, and the individual features are embedded into the D-dimensional space. After stacking the individual features, the color image individual representation matrix x is obtained. c ∈R T×N×D ; 1-2) Represent the color image individual matrix x c ∈R T×N×D Transformed to and 1-3) The features of N individuals in the tth frame Input into the spatial Transformer network, assign importance weights to the relationships between individuals and incorporate them into the characteristics of each individual to reflect the influence of each individual in the group interaction behavior and obtain the individual behavior representation containing spatial interaction information Simplified three-module process, see the following formula: Among them, SPE is spatial position encoding, which is used to supplement the individual position relationship in space; MHSA is a multi-head self-attention mechanism, which is used to infer the spatial interaction of individuals in the scene; FFN is a feedforward network, which is used to further learn the individual interaction in high-dimensional space; LN is a module and performs layer regularization operation; 1-4) The characteristics of the nth individual at T moments Input into the temporal Transformer, and go through the same operation as the three modules in step 2) to obtain the individual behavior representation with cross-frame temporal evolution 1-5) and Transform and fuse to obtain the RGB modality individual representation matrix containing spatiotemporal reasoning 5. The method for group behavior recognition based on cross-modal spatiotemporal self-supervision according to claim 4 is characterized in that: The data processing process of the skeleton modality spatiotemporal interaction network in step (2) is as follows: 2-1): Introduce the high-resolution posture estimation network HRNet to extract the skeleton information of the individual in each frame image, construct the skeleton individual representation matrix corresponding to the original image, and convert the skeleton individual representation matrix x s ∈R T×N×K×3 Transformed to and 2-2): Represent the matrix x for N individuals in the tth frame s ∈R T×N×K×3 Constructing the spatial relationship graph G of human joints t =(V t ,E t ), V t = {v ti |v ti ,i=1,…K joint } and the edge set E t ={e ij |i,j=1,…K joint };v ti Contains node coordinates and confidence, stacked to form e ij Represents the joint node v ti and v tj The relationship between t ; Obtained through spatial graph convolution inference 2-3): Use Construct time node graph G n =(V n ,E n ), where V n = {v ti |v ti ,i=1,…T} and the edge set E n ={e tt′ |t,t′=1,…T,t′≠t}; as in step 2), the spatial graph convolution process is stacked to form And construct the relationship matrix A n ; 2-4): and Transform and fuse to obtain the individual representation matrix containing spatiotemporal skeleton reasoning 2-5): Referring to steps 1-2) to 1-5), the individual representation matrix represented by the spatiotemporal graph convolution Input into the spatiotemporal Transformer for further spatiotemporal relationship reasoning to obtain the skeleton modality individual representation matrix 6. The group behavior recognition method based on cross-modal spatiotemporal self-supervision according to claim 1, characterized in that: The self-supervised multimodal spatiotemporal contrast loss is constructed as follows: (1) The individual representation matrix after multimodal spatiotemporal interactive reasoning and Fusion is performed in spatial and temporal dimensions, eliminating dimensions N and T to obtain RGB modal spatiotemporal interaction representations. and Skeleton modality spatiotemporal interactive representation and (2) Spatial interaction characteristics of the two modes and Apply spatial contrast loss to explore spatial dependencies, consider image modality interaction features and skeleton modality interaction pairs in the same frame as positive samples, and image space interaction features and skeleton space interaction pairs in different frames as negative samples; Taking the tth frame of the image sequence in spatial interaction contrast as an example, the group interaction representation in the tth frame of the skeleton modality is brought closer to increase the similarity, while the individual interaction representations of other frames in the image sequence are excluded to reduce the similarity; therefore, the spatial interaction contrast loss formula is as follows in is the exponent of cosine similarity; 1 [i≠t] ∈{0,1} is an indicator function that is assigned a value of 1 if and only if i≠n; and They are the individual spatial interaction features extracted in the same frame in the image modality and the skeleton modality; (3) Correspondingly, for the temporal interaction characteristics of the two modes and Use temporal contrast loss to explore individual temporal consistency, consider the RGB modality temporal feature and skeleton modality temporal feature pairs of the same individual as positive samples, and consider the RGB modality temporal feature and skeleton modality temporal feature pairs of different individuals as negative samples; Taking the nth individual in the temporal correlation comparison as an example, it is necessary to increase the similarity by pulling in the temporal representation of the nth individual in the skeleton modality, while rejecting the temporal representation of other individuals in the image sequence to reduce the similarity; therefore, the temporal correlation comparison loss is formulated as in and They represent the individual temporal features extracted from the image data and skeleton data of the same individual. In the time dimension, the change of individual features over time also has a significant impact on the evolution of group behavior; (4) The final MSTCo loss is composed of the spatial interaction contrast loss and the temporal contrast correlation loss, and the formula is expressed as: Among them, λ is a hyperparameter used to adjust the weights.

Images