A multi-scale group behavior recognition method, system, and readable storage medium

Abstract

This invention discloses a multi-scale group behavior recognition method, system, and readable storage medium. The method includes: acquiring surveillance video data; inputting the surveillance video data into a preset multi-scale semantic model of group behavior for analysis to obtain key point features of human key point coordinates; analyzing the key point features to obtain the attention mechanism calculation result data of the key point features at the current scale; analyzing the attention mechanism calculation result data of the key point features to obtain a cross-scale representation of the group behavior of the surveillance video data; and analyzing the cross-scale representation of the group behavior to obtain group behavior recognition data. This invention can extract semantic features of group behavior at different scales through a preset model while using a dynamic self-attention mechanism to capture more critical group semantic features. In addition, it also introduces a simple contrastive learning method to assist in learning group behavior.

Description

Technical Field

[0001] This application relates to the fields of data processing and data transmission, and more specifically, to a multi-scale group behavior recognition method, system, and readable storage medium. Background Technology

[0002] Group behavior recognition (GAR) is a task that detects the collective behavior of a group of participants in a video clip. In recent years, GAR has been widely used in video analysis tasks in various contexts such as street corner surveillance videos and sports videos. At present, GAR mainly faces three challenges: (1) Analysis and modeling of groups within a scene. Existing research shows that group behavior is usually represented by the combination of individual behaviors of a few participants in the group. Constructing behavior grouping representations in complex scenes or exploring subgroup segmentation methods in the group is helpful for group behavior analysis; (2) Modeling the interaction relationship between behavioral entities. Existing methods use deep learning to model the interaction relationship of the group and use this to analyze the relative importance between behavioral entities in the group; (3) Multi-scale modeling of the group. Recent research results show that the interaction relationship within the group is multi-scale. Individuals can interact with subgroups, and individuals can also respond to other individuals and produce certain physical behaviors. For example, in volleyball, a player's hitting behavior can cause some or all players in the opposing half of the court to respond, and a player's hand hitting behavior can also cause some or all players in the opposing half of the court to react accordingly. Most existing methods focus on one or two of the challenges mentioned above, studying group behavior by analyzing individual activities of participants or fine group divisions.

[0003] Among existing methods, graph neural network-based approaches are commonly used in group behavior recognition. These methods primarily infer group behavior based on visual features, using participants' visual characteristics as input graph nodes, manually defining interaction relationships between nodes, and using predefined interaction patterns between participants as the graph's adjacency list. Group behavior is represented by aggregating participants' visual features and interaction patterns. While graph-based methods are effective, group behavior occurs in complex scenarios, and simply using undirected or directed graphs is insufficient to fully describe the entire process of group behavior when constructing the graph structure. To define interaction relationships in increasingly complex behavioral scenarios, graph-based methods require the introduction of additional, more sophisticated deep learning methods for adaptation.

[0004] Therefore, the existing technology has defects and urgently needs improvement. Summary of the Invention

[0005] In view of the above problems, the purpose of this invention is to provide a multi-scale group behavior recognition method, system and readable storage medium that can more effectively and quickly identify group behavior.

[0006] The first aspect of this invention provides a multi-scale group behavior recognition method, comprising:

[0007] Acquire surveillance video data;

[0008] The surveillance video data is input into a preset multi-scale semantic model of group behavior for analysis to obtain key point features of human body key point coordinates.

[0009] Based on the keypoint features, the attention mechanism calculation result data X′ of the keypoint features at the current scale is obtained through analysis. si ;

[0010] The result data X′ is calculated based on the attention mechanism of the key point features. si Analysis was performed to obtain a cross-scale representation of the group behavior in the surveillance video data.

[0011] Based on the cross-scale representation of the group behavior The analysis yielded data on group behavior identification.

[0012] This plan also includes:

[0013] Acquire historical image data;

[0014] The historical image data was analyzed to obtain the coordinate features of multiple key points;

[0015] The coordinate features of the multiple key points are integrated and stitched together to obtain the key point features.

[0016] The key point features Mapping to the feature space yields the semantic token X at the keypoint scale. s1 ;

[0017] Obtained by transforming the keypoint token dimension. The dimension of the keypoint token is then reduced to obtain the semantic token X at the individual scale. s2 ;

[0018] Based on the semantic token X at the individual scale s2 Multiple copies are created, and these copies are then copied and concatenated to obtain a semantic token X at the inter-individual interaction scale. s3 ;

[0019] The semantic token X at the keypoint scale s1 Semantic tokens X at the individual scale s2 Semantic token X at the scale of inter-individual interaction s3 By merging, we obtain a multi-scale semantic model X of the pre-defined group behavior.

[0020] In this scheme, the keypoint features are analyzed to obtain the attention mechanism calculation result data X′ of the keypoint features at the current scale. si ,include:

[0021] Based on the key point features of the human body key point coordinates, a semantic token at the current scale is obtained through analysis.

[0022] The semantic token is calculated using the first preset method to obtain the self-attention coefficient of the semantic token at the current scale;

[0023] The self-attention coefficients of the semantic tokens at the current scale are aggregated into the corresponding semantic token features to obtain the self-attention mechanism calculation result data X′ at the current scale. si .

[0024] In this scheme, the attention mechanism based on the key point features calculates the result data X′. si Analysis was performed to obtain a cross-scale representation of the group behavior in the surveillance video data. include:

[0025] The calculation result data X′ of the attention mechanism at the current scale si Each is respectively compared with the X input of the corresponding encoder. si Perform calculations, normalize the results, and send them to the feedforward neural network.

[0026] The calculation result is added to the calculation result of the feedforward neural network and then normalized to obtain a group behavior representation at a single scale.

[0027] Representation of group behavior at the single scale Integration yields a cross-scale representation of group behavior.

[0028] This plan also includes:

[0029] Obtain the token corresponding to the scale;

[0030] The token of the corresponding scale is concatenated with the inference result of the previous encoder, and the concatenated data is converted into cross-scale features.

[0031] The cross-scale features are input into the encoder corresponding to the token for analysis to obtain the cross-scale semantic token X of group behavior. sj .

[0032] In this scheme, the cross-scale representation of the group behavior is used. The analysis yielded group behavior identification data, including:

[0033] Based on the cross-scale representation of the group behavior Analysis was performed to obtain the group behavior feature mapping of the group behavior.

[0034] The group behavior characteristics are mapped according to the second preset method. The analysis yielded cross-scale mapping results of the group behavior;

[0035] Based on the cross-scale mapping results of the group behavior combined with the cross-scale representation of the group behavior Group behavior is identified to obtain group behavior identification data.

[0036] A second aspect of the present invention provides a multi-scale group behavior recognition system, including a memory and a processor. The memory includes a multi-scale group behavior recognition method program, which, when executed by the processor, performs the following steps:

[0037] Acquire surveillance video data;

[0038] The surveillance video data is input into a preset multi-scale semantic model of group behavior for analysis to obtain key point features of human body key point coordinates.

[0039] Based on the keypoint features, the attention mechanism calculation result data X′ of the keypoint features at the current scale is obtained through analysis. si ;

[0040] The result data X′ is calculated based on the attention mechanism of the key point features. si Analysis was performed to obtain a cross-scale representation of the group behavior in the surveillance video data.

[0041] Based on the cross-scale representation of the group behavior The analysis yielded data on group behavior identification.

[0042] This plan also includes:

[0043] Acquire historical image data;

[0044] The historical image data was analyzed to obtain the coordinate features of multiple key points;

[0045] The coordinate features of the multiple key points are integrated and stitched together to obtain the key point features.

[0046] The key point features Mapping to the feature space yields the semantic token X at the keypoint scale. s1 ;

[0047] Obtained by transforming the keypoint token dimension. The dimension of the keypoint token is then reduced to obtain the semantic token X at the individual scale. s2 ;

[0048] Based on the semantic token X at the individual scale s2 Multiple copies are created, and these copies are then copied and concatenated to obtain a semantic token X at the inter-individual interaction scale. s3 ;

[0049] The semantic token X at the keypoint scale s1 Semantic tokens X at the individual scale s2 Semantic token X at the scale of inter-individual interaction s3 By merging, we obtain a multi-scale semantic model X of the pre-defined group behavior.

[0050] In this scheme, the keypoint features are analyzed to obtain the attention mechanism calculation result data X′ of the keypoint features at the current scale. si ,include:

[0051] Based on the key point features of the human body key point coordinates, a semantic token at the current scale is obtained through analysis.

[0052] The semantic token is calculated using the first preset method to obtain the self-attention coefficient of the semantic token at the current scale;

[0053] The self-attention coefficients of the semantic tokens at the current scale are aggregated into the corresponding semantic token features to obtain the self-attention mechanism calculation result data X′ at the current scale. si .

[0054] In this scheme, the attention mechanism based on the key point features calculates the result data X′. si Analysis was performed to obtain a cross-scale representation of the group behavior in the surveillance video data. include:

[0055] The calculation result data X′ of the attention mechanism at the current scale si Each is respectively compared with the X input of the corresponding encoder. si Perform calculations, normalize the results, and send them to the feedforward neural network.

[0056] The calculation result is added to the calculation result of the feedforward neural network and then normalized to obtain a group behavior representation at a single scale.

[0057] Representation of group behavior at the single scale Integration yields a cross-scale representation of group behavior.

[0058] This plan also includes:

[0059] Obtain the token corresponding to the scale;

[0060] The token of the corresponding scale is concatenated with the inference result of the previous encoder, and the concatenated data is converted into cross-scale features.

[0061] The cross-scale features are input into the encoder corresponding to the token for analysis to obtain the cross-scale semantic token X of group behavior. sj .

[0062] In this scheme, the cross-scale representation of the group behavior is used. The analysis yielded group behavior identification data, including:

[0063] Based on the cross-scale representation of the group behavior Analysis was performed to obtain the group behavior feature mapping of the group behavior.

[0064] The group behavior characteristics are mapped according to the second preset method. The analysis yielded cross-scale mapping results of the group behavior;

[0065] Based on the cross-scale mapping results of the group behavior combined with the cross-scale representation of the group behavior Group behavior is identified to obtain group behavior identification data.

[0066] A third aspect of the present invention provides a computer-readable storage medium comprising a multi-scale group behavior recognition method program, wherein when the multi-scale group behavior recognition method program is executed by a processor, it implements the steps of the multi-scale group behavior recognition method as described in any of the preceding claims.

[0067] This invention discloses a multi-scale group behavior recognition method, system, and readable storage medium. The method includes: acquiring surveillance video data; inputting the surveillance video data into a preset multi-scale semantic model of group behavior for analysis to obtain key point features of human key point coordinates; analyzing the key point features to obtain the attention mechanism calculation result data of the key point features at the current scale; analyzing the attention mechanism calculation result data of the key point features to obtain a cross-scale representation of the group behavior of the surveillance video data; and analyzing the cross-scale representation of the group behavior to obtain group behavior recognition data. This invention can extract semantic features of group behavior at different scales through a preset model while using a dynamic self-attention mechanism to capture more critical group semantic features. In addition, it also introduces a simple contrastive learning method to assist in learning group behavior. Attached Figure Description

[0068] Figure 1 A flowchart of a multi-scale group behavior recognition method according to the present invention is shown;

[0069] Figure 2 A flowchart of a method for calculating the attention mechanism results of key point features at the current scale according to the present invention is shown.

[0070] Figure 3 A flowchart of a cross-scale representation calculation method for group behavior in surveillance video data according to the present invention is shown;

[0071] Figure 4 A block diagram of a multi-scale group behavior recognition system according to the present invention is shown;

[0072] Figure 5 The diagram shows the structure of a multi-scale Transformer for group behavior according to the present invention.

[0073] Figure 6 A structural diagram of a Transformer encoder according to the present invention is shown;

[0074] Figure 7 A flowchart illustrating a group behavior recognition method according to the present invention is shown. Detailed Implementation

[0075] To better understand the above-mentioned objectives, features, and advantages of the present invention, the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments. It should be noted that, unless otherwise specified, the embodiments and features described in these embodiments can be combined with each other.

[0076] Many specific details are set forth in the following description in order to provide a full understanding of the invention. However, the invention may also be practiced in other ways different from those described herein, and therefore the scope of protection of the invention is not limited to the specific embodiments disclosed below.

[0077] Figure 1 A flowchart of a multi-scale group behavior recognition method according to the present invention is shown.

[0078] like Figure 1 As shown, this invention discloses a multi-scale group behavior recognition method, comprising:

[0079] S102, Obtain surveillance video data;

[0080] S104, The surveillance video data is input into a preset multi-scale semantic model of group behavior for analysis to obtain key point features of human body key point coordinates;

[0081] S106, Analyze the key point features to obtain the attention mechanism calculation result data of the key point features at the current scale;

[0082] S108, Analyze the data calculated based on the attention mechanism of the key point features to obtain a cross-scale representation of the group behavior of the surveillance video data;

[0083] S110, Analyze the cross-scale representation of the group behavior to obtain group behavior identification data.

[0084] According to an embodiment of the present invention, the multi-scale semantic model of the preset group behavior is a multi-scale dynamic Transformer group behavior recognition model. This model can extract high-level semantic knowledge from group behavior elements represented by the two-dimensional coordinates of human keypoints in a video. The self-attention mechanism in the Transformer encoder learns the self-attention coefficients between tokens to represent their relationships. Based on this, multi-layer perceptrons are used to extract cross-scale features of group behavior across scales. These features are then inferred by multiple Transformer encoders, and a dynamic self-attention mechanism is used to capture more critical group semantic features. Furthermore, a contrastive learning method is introduced to extract group behavior features from the cross-scale representations of group behavior. The group behavior representations obtained from each encoder inference are used as input to a group behavior classifier to assist in learning group behavior.

[0085] According to an embodiment of the present invention, it further includes:

[0086] Acquire historical image data;

[0087] The historical image data was analyzed to obtain the coordinate features of multiple key points;

[0088] The coordinate features of the multiple key points are integrated and stitched together to obtain the key point features.

[0089] The key point features Mapping to the feature space yields the semantic token X at the keypoint scale. s1 ;

[0090] Obtained by transforming the keypoint token dimension. The dimension of the keypoint token is then reduced to obtain the semantic token X at the individual scale. s2 ;

[0091] Based on the semantic token X at the individual scale s2 Multiple copies are created, and these copies are then copied and concatenated to obtain a semantic token X at the inter-individual interaction scale. s3 ;

[0092] The semantic token X at the keypoint scale s1 Semantic tokens X at the individual scale s2 Semantic token X at the scale of inter-individual interaction s3 By merging, we obtain a multi-scale semantic model X of the pre-defined group behavior.

[0093] It should be noted that since human keypoints possess semantic information when segmented and are represented by coordinates in video space, a multi-scale semantic model is constructed based on keypoint coordinates. By employing methods such as position embedding in video space, the keypoint coordinates are converted into semantic feature vectors within the group behavior space. This construction method is suitable for group behavior scenarios with multiple participants and various participating objects. A group behavior semantic model encompassing three scales—keypoints, individuals, and inter-individual interactions—is constructed based on human keypoint coordinates.

[0094] The multiple keypoint coordinate features include a keypoint coordinate time series k, and the keypoint coordinates are processed using regularization. The result of the difference calculation of the coordinates of the key point in the current frame and the next frame. and Keypoint coordinates calculation results embedded in the video spatial and temporal domains and

[0095] First, follow the steps of constructing keypoint tokens. Let the keypoint coordinate time series be... Where N represents the total number of participants in the group, and J represents the number of keypoints per participant. To reduce noise introduced by the pose estimation network when annotating keypoints, the "target keypoint similarity" feature is followed in calculating keypoints. The keypoint coordinates and the difference calculation results between the current frame and the next frame coordinates are then processed using regularization. The results are denoted as follows: and Furthermore, to further extract the spatiotemporal features of keypoint coordinates, keypoint coordinates were embedded along the video spatial domain and the video temporal domain, respectively. The results are denoted as follows: and

[0096] By combining the above calculation results, the key point features are obtained. Where d k The feature dimension of the semantic token is represented by this. Keypoint features are mapped to the feature space via a multilayer perceptron to obtain the semantic token at the keypoint scale.

[0097]

[0098]

[0099] Where ReLU(·) represents the linear rectified function, ReLU(x) = max(0,x), W represents the learnable weights of the hidden layer in the multilayer perceptron, and b represents the bias parameters of the hidden layer in the multilayer perceptron.

[0100] Then, by transforming the keypoint token dimension, we obtain... Using a multilayer perceptron, the keypoint token dimension is reduced from N×(J×d) k ) Dimensionality reduced to N×d k Dimension, thus representing semantic tokens at the individual scale.

[0101]

[0102] Finally, unlike existing studies that use graph-based methods to understand inter-individual interaction patterns, with individual features as graph nodes and inter-individual interaction relationships as graph edges, we follow [1] and construct inter-individual interaction semantic tokens by direct concatenation. This is achieved by creating individual token X. s2 Two copies and Duplicate the individual token N times to obtain and and transpose the copy Later and copy The concatenation process, followed by embedding the result into the feature space using a multilayer perceptron, serves as a semantic token X for inter-individual interaction. s3 .

[0103]

[0104] In summary, by merging the tokens obtained from each scale, we arrive at a multi-scale semantic model X for group behavior:

[0105] X = {X si |i=1,2,3}

[0106] Figure 2 A flowchart of a method for calculating the attention mechanism results of key point features at the current scale according to the present invention is shown.

[0107] like Figure 2 As shown in the embodiment of the present invention, the attention mechanism calculation result data X′ of the keypoint features at the current scale is obtained by analyzing the keypoint features. si ,include:

[0108] S202, Analyze the key point features based on the coordinates of the human body key points to obtain the semantic token at the current scale;

[0109] S204, The semantic token is calculated using the first preset method to obtain the self-attention coefficient of the semantic token at the current scale;

[0110] S206, aggregate the self-attention coefficients of the semantic tokens at the current scale into the corresponding semantic token features to obtain the self-attention mechanism calculation result data X′ at the current scale. si .

[0111] It should be noted that the first preset method is the scaled dot product method. In this scheme, the multi-scale Transformer uses an encoder to infer group behavior features at each scale. First, in order to obtain the group behavior representation at a single-scale token inference, the Transformer encoder is used to embed the tokens separately, and the self-attention coefficients of the tokens are calculated by the scaled dot product method. Matrix multiplication aggregates the self-attention coefficients into the semantic token features to represent the group behavior features at that scale.

[0112] Q si =Linear(X) si ),K si =Linear(X) si ),V si =Linear(X) si )

[0113]

[0114] Among them, Attentionsi (·) represents the self-attention mechanism, Linear(·) represents the linear layer, and here it represents the embedding operation, with Linear(x) = xW. Q si ,K si V si This represents the query matrix and a key-value pair obtained from the semantic token embedding operation, with softmax(·) denoteing the normalized exponential function. Self-attention mechanism (Attention) si The result of (·) is X′ si .

[0115] Figure 3 A flowchart of a cross-scale representation calculation method for group behavior in surveillance video data according to the present invention is shown.

[0116] like Figure 3 As shown in the embodiment of the present invention, the attention mechanism based on the key point features calculates the result data X′. si Analysis was performed to obtain a cross-scale representation of the group behavior in the surveillance video data. include:

[0117] S302, the attention mechanism calculation result data X′ at the current scale is... si Each is respectively compared with the X input of the corresponding encoder. si Perform calculations, normalize the results, and send them to the feedforward neural network.

[0118] S304, the calculation result is added to the calculation result of the feedforward neural network and then normalized to obtain a group behavior representation at a single scale.

[0119] S306, Representation of group behavior at the single scale Integration yields a cross-scale representation of group behavior.

[0120] It should be noted that after obtaining the calculation results of the attention mechanism, X′ si The result is compared with the encoder input X. si The addition operation, after layer normalization, is fed into the feedforward neural network. The result is added again to the result calculated by the feedforward neural network, and then normalized again to obtain the group behavior representation at a single scale. The specific calculation formula for the layer normalization operation is X″. si =LN(X′) si +X si ).

[0121] According to an embodiment of the present invention, it further includes:

[0122] Obtain the token corresponding to the scale;

[0123] The token of the corresponding scale is concatenated with the inference result of the previous encoder, and the concatenated data is converted into cross-scale features.

[0124] The cross-scale features are input into the encoder corresponding to the token for analysis to obtain the cross-scale semantic token X of group behavior. sj .

[0125] It should be noted that, in order to further explore the inter-scale relationships of multi-scale group behavior semantics, the tokens of the corresponding scale are concatenated with the inference results of the previous encoder before the second and third encoders of the Transformer. After being embedded into cross-scale features by a multilayer perceptron, the resulting cross-scale tokens are fed into the current scale encoder to obtain the cross-scale semantic tokens of group behavior, denoted as X. sj :

[0126]

[0127] According to an embodiment of the present invention, it further includes:

[0128] The cross-scale representation based on the group behavior The analysis yielded group behavior identification data, including:

[0129] Based on the cross-scale representation of the group behavior Analysis was performed to obtain the group behavior feature mapping of the group behavior.

[0130] The group behavior characteristics are mapped according to the second preset method. The analysis yielded cross-scale mapping results of the group behavior;

[0131] Based on the cross-scale mapping results of the group behavior combined with the cross-scale representation of the group behavior Group behavior is identified to obtain group behavior identification data.

[0132] It should be noted that the second preset method is the Barlow Twins method. Based on the multi-scale semantic model and the multi-scale Transformer, a cross-scale representation of group behavior is finally obtained. in Includes only keypoint scale information. It contains cross-scale information at the keypoint and individual scales. It contains multi-scale information about group behavior. These inference results at different scales should have the same group behavior representation. By introducing a contrastive learning method, group behavior features are extracted from the cross-scale representation of group behavior, and the group behavior representation obtained by each encoder is used as the input to the group behavior classifier.

[0133] First, in order to improve the encoder output at each scale, the representation obtained by the encoder is... The group behavior feature mapping is obtained through a global pooling layer consisting of a normalization layer, an activation layer, and a max pooling layer, followed by a linear layer. Where d s B represents the output dimension of the linear layer, and B represents the batch size during network training.

[0134] Then, in order to obtain higher-level group behavior features between the cross-scale representations of group behavior, the Barlow Twins method is used to cluster the cross-scale mapping results to assist in group behavior recognition, taking the group behavior mapping as input.

[0135]

[0136] Among them, L BT The specific calculation method for (·) is as follows:

[0137]

[0138] in, λ represents the covariance matrix of the group behavior mapping under the current training batch size, obtained by matrix multiplication of the group behavior mapping. mn C represents the scaling factor of the trace of the covariance matrix. nm The elements within the covariance matrix are represented by the following method for calculation:

[0139]

[0140]

[0141] According to an embodiment of the present invention, it further includes:

[0142] Augmented data is obtained by processing historical data through methods such as random masking of participants, horizontal flipping of video frames, and overall translation of the coordinates of key points of participants;

[0143] The augmented data is used to assist in training the multi-scale semantic model X of the preset group behavior, thereby improving the generalization ability of the model.

[0144] It should be noted that this solution employs data augmentation methods commonly used in group behavior recognition to augment the keypoint coordinates, thereby assisting model training and improving the model's generalization ability. These data augmentation methods include random masking of participants, horizontal flipping of video frames, and overall translation of participant keypoint coordinates.

[0145] Participants are randomly masked. Within the input video clip, a participant is randomly selected based on the order of their annotations, and all their keypoint coordinates are set to zero.

[0146] Horizontal frame flipping. Horizontally flipping all video frames within an input video segment is a common data augmentation technique in group behavior recognition tasks. This augmentation method horizontally flips the pose and behavior of each participant in the video, and also adds perturbations to each flipped keypoint.

[0147] The keypoint coordinates of the participants are translated as a whole. Similar to random masking of participants, within the input video clip, a participant is randomly selected based on the order of their annotations, and their keypoints are moved vertically and / or horizontally by a certain number of pixels. The horizontal movement distance is limited to half the length of the video frame, and the vertical movement distance is limited to half the width of the video frame.

[0148] Figure 4 A block diagram of a multi-scale group behavior recognition system according to the present invention is shown.

[0149] like Figure 4 As shown, a second aspect of the present invention provides a multi-scale group behavior recognition system 4, including a memory 41 and a processor 42. The memory includes a multi-scale group behavior recognition method program, which, when executed by the processor, performs the following steps:

[0150] Acquire surveillance video data;

[0151] The surveillance video data is input into a preset multi-scale semantic model of group behavior for analysis to obtain key point features of human body key point coordinates.

[0152] Based on the key point features, the attention mechanism calculation results of the key point features at the current scale are obtained through analysis.

[0153] The results of the attention mechanism calculation based on the key point features are analyzed to obtain a cross-scale representation of the group behavior of the surveillance video data.

[0154] Group behavior identification data is obtained by analyzing the cross-scale representation of the group behavior.

[0155] According to an embodiment of the present invention, the multi-scale semantic model of the preset group behavior is a multi-scale dynamic Transformer group behavior recognition model. This model can extract high-level semantic knowledge from group behavior elements represented by the two-dimensional coordinates of human keypoints in a video. The self-attention mechanism in the Transformer encoder learns the self-attention coefficients between tokens to represent their relationships. Based on this, multi-layer perceptrons are used to extract cross-scale features of group behavior across scales. These features are then inferred by multiple Transformer encoders, and a dynamic self-attention mechanism is used to capture more critical group semantic features. Furthermore, a contrastive learning method is introduced to extract group behavior features from the cross-scale representations of group behavior. The group behavior representations obtained from each encoder inference are used as input to a group behavior classifier to assist in learning group behavior.

[0156] According to an embodiment of the present invention, it further includes:

[0157] Acquire historical image data;

[0158] The historical image data was analyzed to obtain the coordinate features of multiple key points;

[0159] The coordinate features of the multiple key points are integrated and stitched together to obtain the key point features.

[0160] The key point features Mapping to the feature space yields the semantic token X at the keypoint scale. s1 ;

[0161] Obtained by transforming the keypoint token dimension. The dimension of the keypoint token is then reduced to obtain the semantic token X at the individual scale. s2 ;

[0162] Based on the semantic token X at the individual scale s2 Multiple copies are created, and these copies are then copied and concatenated to obtain a semantic token X at the inter-individual interaction scale. s3 ;

[0163] The semantic token X at the keypoint scale s1 Semantic tokens X at the individual scale s2 Semantic token X at the scale of inter-individual interaction s3 By merging, we obtain a multi-scale semantic model X of the pre-defined group behavior.

[0164] It should be noted that since human keypoints possess semantic information when segmented and are represented by coordinates in video space, a multi-scale semantic model is constructed based on keypoint coordinates. By employing methods such as position embedding in video space, the keypoint coordinates are converted into semantic feature vectors within the group behavior space. This construction method is suitable for group behavior scenarios with multiple participants and various participating objects. A group behavior semantic model encompassing three scales—keypoints, individuals, and inter-individual interactions—is constructed based on human keypoint coordinates.

[0165] The multiple keypoint coordinate features include a keypoint coordinate time series k, and the keypoint coordinates are processed using regularization. The result of the difference calculation of the coordinates of the key point in the current frame and the next frame. and Keypoint coordinates calculation results embedded in the video spatial and temporal domains and

[0166] First, follow the steps of constructing keypoint tokens. Let the keypoint coordinate time series be... Where N represents the total number of participants in the group, and J represents the number of keypoints per participant. To reduce noise introduced by the pose estimation network when annotating keypoints, the "target keypoint similarity" feature is followed in calculating keypoints. The keypoint coordinates and the difference calculation results between the current frame and the next frame coordinates are then processed using regularization. The results are denoted as follows: and Furthermore, to further extract the spatiotemporal features of keypoint coordinates, keypoint coordinates were embedded along the video spatial domain and the video temporal domain respectively, and the results are denoted as follows: and

[0167] By combining the above calculation results, the key point features are obtained. Where d k The feature dimension of the semantic token is represented by this. Keypoint features are mapped to the feature space via a multilayer perceptron to obtain the semantic token at the keypoint scale.

[0168]

[0169]

[0170] Where ReLU(·) represents the linear rectified function, ReLU(x) = max(0,x), W represents the learnable weights of the hidden layer in the multilayer perceptron, and b represents the bias parameters of the hidden layer in the multilayer perceptron.

[0171] Then, by transforming the keypoint token dimension, we obtain... Using a multilayer perceptron, the keypoint token dimension is reduced from N×(J×d) k ) Dimensionality reduced to N×d k Dimension, thus representing semantic tokens at the individual scale.

[0172]

[0173] Finally, unlike existing studies that use graph-based methods to understand inter-individual interaction patterns, with individual features as graph nodes and inter-individual interaction relationships as graph edges, we follow [1] and construct inter-individual interaction semantic tokens by direct concatenation. This is achieved by creating individual token X. s2 Two copies and Duplicate the individual token N times to obtain and and transpose the copy Later and copy The concatenation process, followed by embedding the result into the feature space using a multilayer perceptron, serves as a semantic token X for inter-individual interaction. s3 .

[0174]

[0175] In summary, by merging the tokens obtained from each scale, we arrive at a multi-scale semantic model X for group behavior:

[0176] X = {X si |i=1,2,3}

[0177] According to an embodiment of the present invention, the analysis based on the keypoint features yields the attention mechanism calculation result data X′ of the keypoint features at the current scale. si ,include:

[0178] Based on the key point features of the human body key point coordinates, a semantic token at the current scale is obtained through analysis.

[0179] The semantic token is calculated using the first preset method to obtain the self-attention coefficient of the semantic token at the current scale;

[0180] The self-attention coefficients of the semantic tokens at the current scale are aggregated into the corresponding semantic token features to obtain the self-attention mechanism calculation result data X′ at the current scale. si .

[0181] It should be noted that the first preset method is the scaled dot product method. In this scheme, the multi-scale Transformer uses an encoder to infer group behavior features at each scale. First, in order to obtain the group behavior representation at a single-scale token inference, the Transformer encoder is used to embed the tokens separately, and the self-attention coefficients of the tokens are calculated by the scaled dot product method. Matrix multiplication aggregates the self-attention coefficients into the semantic token features to represent the group behavior features at that scale.

[0182] Q si =Linear(X) si ),K si =Linear(X) si ),V si =Linear(X) si )

[0183]

[0184] Among them, Attention si (·) represents the self-attention mechanism, Linear(·) represents the linear layer, and here it represents the embedding operation, with Linear(x) = xW. Q si ,K si V si This represents the query matrix and a key-value pair obtained from the semantic token embedding operation, with softmax(·) denoteing the normalized exponential function. Self-attention mechanism (Attention) si The result of (·) is X′ si .

[0185] According to an embodiment of the present invention, the attention mechanism based on the key point features calculates the result data X′. si Analysis was performed to obtain a cross-scale representation of the group behavior in the surveillance video data. include:

[0186] The calculation result data X′ of the attention mechanism at the current scale si Each is respectively compared with the X input of the corresponding encoder. si Perform calculations, normalize the results, and send them to the feedforward neural network.

[0187] The calculation result is added to the calculation result of the feedforward neural network and then normalized to obtain a group behavior representation at a single scale.

[0188] Representation of group behavior at the single scale Integration yields a cross-scale representation of group behavior.

[0189] It should be noted that after obtaining the calculation results of the attention mechanism, X′ si The result is compared with the encoder input X. si The addition operation, after layer normalization, is fed into the feedforward neural network. The result is added again to the result calculated by the feedforward neural network, and then normalized again to obtain the group behavior representation at a single scale. The specific calculation formula for the layer normalization operation is X″. si =LN(X′) si +X si ).

[0190] According to an embodiment of the present invention, it further includes:

[0191] Obtain the token corresponding to the scale;

[0192] The token of the corresponding scale is concatenated with the inference result of the previous encoder, and the concatenated data is converted into cross-scale features.

[0193] The cross-scale features are input into the encoder corresponding to the token for analysis to obtain the cross-scale semantic token X of group behavior. sj .

[0194] It should be noted that, in order to further explore the inter-scale relationships of multi-scale group behavior semantics, the tokens of the corresponding scale are concatenated with the inference results of the previous encoder before the second and third encoders of the Transformer. After being embedded into cross-scale features by a multilayer perceptron, the resulting cross-scale tokens are fed into the current scale encoder to obtain the cross-scale semantic tokens of group behavior, denoted as X. sj :

[0195]

[0196] According to an embodiment of the present invention, it further includes:

[0197] The cross-scale representation based on the group behavior The analysis yielded group behavior identification data, including:

[0198] Based on the cross-scale representation of the group behavior Analysis was performed to obtain the group behavior feature mapping of the group behavior.

[0199] The group behavior characteristics are mapped according to the second preset method. The analysis yielded cross-scale mapping results of the group behavior;

[0200] Based on the cross-scale mapping results of the group behavior combined with the cross-scale representation of the group behavior Group behavior is identified to obtain group behavior identification data.

[0201] It should be noted that the second preset method is the Barlow Twins method. Based on the multi-scale semantic model and the multi-scale Transformer, a cross-scale representation of group behavior is finally obtained. in Includes only keypoint scale information. It contains cross-scale information at the keypoint and individual scales. It contains multi-scale information about group behavior. These inference results at different scales should have the same group behavior representation. By introducing a contrastive learning method, group behavior features are extracted from the cross-scale representation of group behavior, and the group behavior representation obtained by each encoder is used as the input to the group behavior classifier.

[0202] First, in order to improve the encoder output at each scale, the representation obtained by the encoder is... The group behavior feature mapping is obtained through a global pooling layer consisting of a normalization layer, an activation layer, and a max pooling layer, followed by a linear layer. Where d s B represents the output dimension of the linear layer, and B represents the batch size during network training.

[0203] Then, in order to obtain higher-level group behavior features between the cross-scale representations of group behavior, the Barlow Twins method is used to cluster the cross-scale mapping results to assist in group behavior recognition, taking the group behavior mapping as input.

[0204]

[0205] Among them, L BT The specific calculation method for (·) is as follows:

[0206]

[0207] in, λ represents the covariance matrix of the group behavior mapping under the current training batch size, obtained by matrix multiplication of the group behavior mapping. mn C represents the scaling factor of the trace of the covariance matrix. nm The elements within the covariance matrix are represented by the following method for calculation:

[0208]

[0209]

[0210] According to an embodiment of the present invention, it further includes:

[0211] Augmented data is obtained by processing historical data through methods such as random masking of participants, horizontal flipping of video frames, and overall translation of the coordinates of key points of participants;

[0212] The augmented data is used to assist in training the multi-scale semantic model X of the preset group behavior, thereby improving the generalization ability of the model.

[0213] It should be noted that this solution employs data augmentation methods commonly used in group behavior recognition to augment the keypoint coordinates, thereby assisting model training and improving the model's generalization ability. These data augmentation methods include random masking of participants, horizontal flipping of video frames, and overall translation of participant keypoint coordinates.

[0214] Participants are randomly masked. Within the input video clip, a participant is randomly selected based on the order of their annotations, and all their keypoint coordinates are set to zero.

[0215] Horizontal frame flipping. Horizontally flipping all video frames within an input video segment is a common data augmentation technique in group behavior recognition tasks. This augmentation method horizontally flips the pose and behavior of each participant in the video, and also adds perturbations to each flipped keypoint.

[0216] The keypoint coordinates of the participants are translated as a whole. Similar to random masking of participants, within the input video clip, a participant is randomly selected based on the order of their annotations, and their keypoints are moved vertically and / or horizontally by a certain number of pixels. The horizontal movement distance is limited to half the length of the video frame, and the vertical movement distance is limited to half the width of the video frame.

[0217] This invention discloses a multi-scale group behavior recognition method, system, and readable storage medium. The method includes: acquiring surveillance video data; inputting the surveillance video data into a preset multi-scale semantic model of group behavior for analysis to obtain key point features of human key point coordinates; analyzing the key point features to obtain the attention mechanism calculation result data of the key point features at the current scale; analyzing the attention mechanism calculation result data of the key point features to obtain a cross-scale representation of the group behavior of the surveillance video data; and analyzing the cross-scale representation of the group behavior to obtain group behavior recognition data. This invention can extract semantic features of group behavior at different scales through a preset model while using a dynamic self-attention mechanism to capture more critical group semantic features. In addition, it also introduces a simple contrastive learning method to assist in learning group behavior.

[0218] In the several embodiments provided in this application, it should be understood that the disclosed devices and methods can be implemented in other ways. The device embodiments described above are merely illustrative. For example, the division of units is only a logical functional division, and in actual implementation, there may be other division methods, such as: multiple units or components can be combined, or integrated into another system, or some features can be ignored or not executed. In addition, the coupling, direct coupling, or communication connection between the various components shown or discussed can be through some interfaces, and the indirect coupling or communication connection between devices or units can be electrical, mechanical, or other forms.

[0219] The units described above as separate components may or may not be physically separate. The components shown as units may or may not be physical units. They may be located in one place or distributed across multiple network units. Some or all of the units may be selected to achieve the purpose of this embodiment according to actual needs.

[0220] In addition, in the various embodiments of the present invention, each functional unit can be integrated into one processing unit, or each unit can be a separate unit, or two or more units can be integrated into one unit; the integrated unit can be implemented in hardware or in the form of hardware plus software functional units.

[0221] Those skilled in the art will understand that all or part of the steps of the above method embodiments can be implemented by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When the program is executed, it performs the steps of the above method embodiments. The aforementioned storage medium includes various media capable of storing program code, such as mobile storage devices, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0222] Alternatively, if the integrated units of this invention are implemented as software functional modules and sold or used as independent products, they can also be stored in a computer-readable storage medium. Based on this understanding, the technical solutions of the embodiments of this invention, or the parts that contribute to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the methods described in the various embodiments of this invention. The aforementioned storage medium includes various media capable of storing program code, such as mobile storage devices, ROM, RAM, magnetic disks, or optical disks.

Claims

1. A multi-scale group behavior recognition method, characterized in that, include: Acquire surveillance video data; The surveillance video data is input into a preset multi-scale semantic model of group behavior for analysis to obtain key point features of human body key point coordinates. Based on the keypoint features, the attention mechanism calculation results of the keypoint features at the current scale are obtained through analysis. ; The data is calculated based on the attention mechanism of the key point features. Analysis was performed to obtain a cross-scale representation of the group behavior in the surveillance video data. ; Based on the cross-scale representation of the group behavior The analysis yielded group behavior identification data; Acquire historical image data; The historical image data was analyzed to obtain the coordinate features of multiple key points; The coordinate features of the multiple key points are integrated and stitched together to obtain the key point features. ; The key point features Mapping to the feature space yields semantic tokens at the keypoint scale. ; Obtained by transforming the keypoint token dimension. Furthermore, the dimensionality of the keypoint tokens is reduced to obtain semantic tokens at the individual scale. ; Based on the semantic tokens at the individual scale Multiple copies are created, and these copies are replicated and concatenated to obtain a semantic token at the inter-individual interaction scale. ; semantic tokens at the keypoint scale Semantic tokens at the individual scale Semantic tokens at the scale of inter-individual interaction By merging, we obtain a multi-scale semantic model X of the pre-defined group behavior.

2. The multi-scale group behavior recognition method according to claim 1, characterized in that, The analysis based on the keypoint features yields the attention mechanism calculation results data for the keypoint features at the current scale. ,include: Based on the key point features of the human body key point coordinates, a semantic token at the current scale is obtained through analysis. The semantic token is calculated using the first preset method to obtain the self-attention coefficient of the semantic token at the current scale; The self-attention coefficients of the semantic tokens at the current scale are aggregated into the corresponding semantic token features to obtain the self-attention mechanism calculation results data at the current scale. .

3. The multi-scale group behavior recognition method according to claim 1, characterized in that, The data calculated based on the attention mechanism of the key point features Analysis was performed to obtain a cross-scale representation of the group behavior in the surveillance video data. ,include: The data calculated using the attention mechanism at the current scale Each with the corresponding encoder input Perform calculations, normalize the results, and send them to the feedforward neural network. The calculation result is added to the calculation result of the feedforward neural network and then normalized to obtain a group behavior representation at a single scale. ; Representation of group behavior at the single scale Integration yields a cross-scale representation of group behavior. .

4. The multi-scale group behavior recognition method according to claim 1, characterized in that, Also includes: Obtain the token corresponding to the scale; The token of the corresponding scale is concatenated with the inference result of the previous encoder, and the concatenated data is converted into cross-scale features. The cross-scale features are input into the encoder corresponding to the token for analysis to obtain cross-scale semantic tokens of group behavior. .

5. The multi-scale group behavior recognition method according to claim 1, characterized in that, The cross-scale representation based on the group behavior The analysis yielded group behavior identification data, including: Based on the cross-scale representation of the group behavior Analysis was performed to obtain the group behavior feature mapping of the group behavior. ; The group behavior characteristics are mapped according to the second preset method. The analysis yielded cross-scale mapping results of the group behavior; Based on the cross-scale mapping results of the group behavior combined with the cross-scale representation of the group behavior Group behavior is identified to obtain group behavior identification data.

6. A multi-scale group behavior recognition system, characterized in that, The system includes a memory and a processor. The memory contains a multi-scale group behavior recognition method program, which, when executed by the processor, performs the following steps: Acquire surveillance video data; The surveillance video data is input into a preset multi-scale semantic model of group behavior for analysis to obtain key point features of human body key point coordinates. Based on the keypoint features, the attention mechanism calculation results of the keypoint features at the current scale are obtained through analysis. ; The data is calculated based on the attention mechanism of the key point features. Analysis was performed to obtain a cross-scale representation of the group behavior in the surveillance video data. ; Based on the cross-scale representation of the group behavior The analysis yielded group behavior identification data; Acquire historical image data; The historical image data was analyzed to obtain the coordinate features of multiple key points; The coordinate features of the multiple key points are integrated and stitched together to obtain the key point features. ; The key point features Mapping to the feature space yields semantic tokens at the keypoint scale. ; Obtained by transforming the keypoint token dimension. Furthermore, the dimensionality of the keypoint tokens is reduced to obtain semantic tokens at the individual scale. ; Based on the semantic tokens at the individual scale Multiple copies are created, and these copies are replicated and concatenated to obtain a semantic token at the inter-individual interaction scale. ; semantic tokens at the keypoint scale Semantic tokens at the individual scale Semantic tokens at the scale of inter-individual interaction By merging, we obtain a multi-scale semantic model X of the pre-defined group behavior.

7. A multi-scale group behavior recognition system according to claim 6, characterized in that, The analysis based on the keypoint features yields the attention mechanism calculation results data for the keypoint features at the current scale. ,include: Based on the key point features of the human body key point coordinates, a semantic token at the current scale is obtained through analysis. The semantic token is calculated using the first preset method to obtain the self-attention coefficient of the semantic token at the current scale; The self-attention coefficients of the semantic tokens at the current scale are aggregated into the corresponding semantic token features to obtain the self-attention mechanism calculation results data at the current scale. .

8. A multi-scale group behavior recognition system according to claim 7, characterized in that, The data calculated based on the attention mechanism of the key point features Analysis was performed to obtain a cross-scale representation of the group behavior in the surveillance video data. ,include: The data calculated using the attention mechanism at the current scale Each with the corresponding encoder input Perform calculations, normalize the results, and send them to the feedforward neural network. The calculation result is added to the calculation result of the feedforward neural network and then normalized to obtain a group behavior representation at a single scale. ; Representation of group behavior at the single scale Integration yields a cross-scale representation of group behavior. .

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium includes a multi-scale group behavior recognition method program, which, when executed by a processor, implements the steps of the multi-scale group behavior recognition method as described in any one of claims 1 to 5.

Images