Group behavior identification method and system based on cross-feature interaction Transform

Abstract

The invention discloses a group behavior identification method and system based on cross-feature interaction Transform, and the method comprises the steps: firstly extracting the appearance and posture dual-branch features of a video clip, inputting the features into a cross-feature correction module for coding interaction, and generating a correction vector through pooling splicing and MLP to achieve feature optimization; a correction result is input into a cross-feature interaction Transform module; spatial information and gating bottleneck block dynamic calibration features are integrated through position coding, and interactive perception features are generated by using a cross attention mechanism; the features are integrated with spatio-temporal information through an asymmetric convolution fusion module, and standard symmetric convolution is replaced to enhance local details and global context; and finally, through full-connection layer dimension transformation, the Softmax layer outputs probability distribution of group behavior categories. Through three innovations of feature correction, interactive Transform and asymmetric fusion, complementary information of multi-modal features is effectively mined, and the recognition precision is remarkably improved.

Description

Technical Field

[0001] The present invention belongs to the technical field of group behavior recognition, and particularly relates to a group behavior recognition method and system based on cross-feature interaction Transformer. Background Art

[0002] In the past decade, group behavior recognition, as a mainstream research branch of human behavior recognition, has shown important application potential in many fields such as intelligent traffic management, animal behavior research, and autonomous driving. With the development of large-scale datasets and deep learning algorithms, group behavior recognition has made relatively obvious progress. However, most existing group behavior recognition methods only use the single visual feature of RGB frames as input, or adopt simple fusion strategies to integrate parallel feature streams. Such methods limit the feature representation ability of the model and are difficult to provide sufficient and complementary information. To overcome these limitations, the present invention focuses on deeply mining the complementary information in cross-features and achieving accurate group behavior recognition through an interactive fusion method.

[0003] Deep learning technology has achieved remarkable success in many fields of computer vision, such as image recognition, object detection, semantic segmentation, etc., and has further promoted the development of the GAR field. Early research combined convolutional neural networks and recurrent neural networks to simulate the temporal dynamics of groups and used various attention mechanism-based methods to extract key actors highly related to group behaviors. Although these methods have achieved certain results, they have not fully considered the long-term temporal dependence in group behavior recognition. In recent years, researchers have begun to explore the application of graph convolutional networks and Transformer models in group behavior recognition. These methods significantly improve the accuracy of group behavior recognition by modeling the dynamic spatio-temporal interaction relationships between individual actors. For example, ARG constructs an actor relationship graph to automatically update the feature representations of individual actors by fusing graph convolutional networks and sparse time sampling methods, thereby learning the global representation of group behaviors. Actor-Transformer first introduced the Transformer model into the group behavior recognition task and effectively processed the spatio-temporal information in video data by separately modeling spatial and temporal information. At the same time, subsequent variant studies based on the Transformer model unifiedly modeled spatio-temporal information through simple feature fusion schemes to simulate the spatio-temporal interaction between multiple actors. However, most existing methods still adopt early or late fusion strategies to integrate different feature branches, resulting in the complex interaction relationships between different feature streams not being fully learned. Therefore, future research needs to explore the interaction between different feature streams to provide meaningful complementary information and mine higher-quality feature representations.

[0004] Although group behavior recognition has received increasing attention in recent years, there are still some deficiencies in current GAR methods in terms of inferring interaction relationships, mainly because the complex interactions between feature streams have not been fully modeled in depth. Some scholars have applied a 2D pose network to single-frame images and a 3D CNN to all input frames simultaneously to capture the rich static features and dynamic features of individual actors respectively. However, the simple fusion strategy adopted by this method may not be sufficient to fully capture the complex interactions of group behaviors, because it fails to make full use of the complementarity between different feature streams, limiting the model's ability to understand and infer interaction relationships at a deeper level. Other scholars have introduced three different fusion strategies to integrate the pose branch and the global branch, and this design helps to establish a connection between local dynamic features and global context information to achieve better group behavior recognition performance. However, this method does not fully utilize the complementary advantages of different feature streams in characterizing group behaviors, indicating that there is still room for further improvement in feature fusion and utilization.

[0005] Based on the above discussion and analysis, the present invention proposes an innovative group behavior recognition method and system based on Cross-Feature Interaction Transformer (CFI-Former). Summary of the Invention

[0006] To solve the above technical problems, the present invention provides a group behavior recognition method and system based on Cross-Feature Interaction Transformer.

[0007] The technical solution adopted by the present invention to solve its technical problems is as follows: A group behavior recognition method based on Cross-Feature Interaction Transformer, the method comprising the following steps: S100: Obtain a video segment and input it into a feature extraction network, and extract features from the input video frames through appearance and pose dual-branch feature streams to obtain appearance features and pose features; S200: Input the appearance features and pose features into a cross-feature correction module, encode the appearance features and pose features, perform feature interaction through pooling and splicing operations, then generate and split vectors through an MLP, and finally output a cross-feature correction result through channel-level multiplication correction; S300: Input the cross-feature correction result into a cross-feature interaction Transformer module, integrate spatial position information using position encoding, calibrate dynamic features through a gated bottleneck block, then generate queries, keys, and values through linear projection, and generate interaction-aware features for group behavior recognition in combination with the cross-attention mechanism; S400: Input the interaction perception features into the feature fusion module, and perform fusion through asymmetric convolution to obtain the fused feature representation; S500: Input the fused feature representation into the fully connected layer for dimensional transformation to match the number of group behavior categories. After passing through the Softmax layer, convert the output of the fully connected layer into a probability distribution, and determine the confidence of each group behavior category, thereby completing the recognition of group behaviors.

[0008] Preferably, S100 includes: S110: Uniformly sample frame videos from the video clips. Each video frame contains actor bounding boxes, denoted as , where represent the height and width of the video frame respectively; S120: In the pose branch, given the input video frame and the individual bounding box, use HRNet as the pose estimation network to extract the two-dimensional joint coordinates of the individual , where 17 represents the number of predicted joints; S130: Generate the pose feature vector through the fully connected layer for the joint coordinates , where represents the dimension of the pose feature vector; S140: In the appearance branch, given the input video frame and the individual bounding box, use the Inception-v3 feature extraction network as the backbone network to generate a high-dimensional feature map. Use the RoIAlign operation to crop the corresponding region on the feature map according to the bounding box coordinates, and adjust the region to a unified size. Then, compress the aligned feature map into a feature vector through the average pooling operation to retain the appearance information of the individual actor. The appearance feature of the individual actor is represented as , where represents the dimension of the appearance feature vector.

[0009] Preferably, S200 includes: S210: After extracting the pose features and appearance features , use a 1×1 convolutional layer combined with the ReLU activation function to convert the individual features into -dimensional pose feature vectors and appearance feature vectors ; S220: Along the channel dimension, perform global max pooling and global average pooling operations on both and . Concatenate the four resulting vectors obtained through the above pooling operations to form the concatenated feature vector , to explore the interaction between appearance features and pose features; where C represents the dimension of the feature vector; S230: Use a multi-layer perceptron MLP network with a Sigmoid activation function to obtain the final vector representation from and further split it into and and ; S240: Multiply and separately with and at the channel level to obtain the final cross-feature channel-level corrected pose feature vector and the corrected appearance feature vector .

[0010] Preferably, S210 is specifically: , ; Among them, represents the pose feature vector after being processed by 1×1 convolution and ReLU activation function, represents the appearance feature vector after being transformed in the channel dimension and activated by ReLU; S230 is specifically: ; Among them, represents the concatenated feature vector, represents the multi-layer perceptron MLP network, represents the correction weight of the pose feature, represents the correction weight of the appearance feature; S240 is specifically: ; Among them, represents channel-level multiplication, performing element-wise multiplication on the elements of each channel in the feature map, represents the corrected pose feature vector, represents the corrected appearance feature vector.

[0011] Preferably, in S300, position encoding is used to integrate spatial position information, dynamic feature calibration is achieved through a gated bottleneck block, and then queries, keys, and values are generated through linear projection, including: S310: Use position encoding to integrate spatial position information and fuse it into the output representation of the cross-feature correction module; S320: The gated bottleneck block receives the and as input, and learn the element-wise weight mapping respectively and , thereby being able to dynamically adjust the information flow between feature streams, and are calculated as follows: ; ; wherein, represents an addition operation in the channel dimension; S330: The output of the gated bottleneck block is expressed as: ; wherein, represents an element-wise dot product operation; S340: After the gated bottleneck block, the output features of the gated bottleneck block are linearly projected using learnable weight parameters to generate query , key and value respectively, and query , key and value , and the calculation process is expressed as: ; ; wherein, , and represent learnable weight parameters.

[0012] Preferably, in S300, an interactive perception feature for group behavior recognition is generated in combination with a cross-attention mechanism, including: S350: Using a scaled dot-product attention mechanism, calculate the global attention map between and , then aggregate it with value , and calculate the global attention map between and , then aggregate it with value to obtain an interactive perception feature representation and , and the calculation process of its cross-attention is expressed as: ; wherein, the softmax function is used to calculate the similarity between query and key , and calculate the similarity between query and key , and dim represents the dimension of the query vector and the key vector; S360: To enhance the training stability of the model, the corrected pose feature vector after integrating the positional encoding and the appearance feature vector are respectively added to the pose interaction perception feature and the appearance interaction perception feature output by the cross-attention. Subsequently, layer normalization operations are performed to stabilize the training process. Finally, the features are further transformed through a feed-forward neural network to generate a more fine-grained interaction perception feature representation and .

[0013] Preferably, S400 includes: S410: Perform an addition operation on the more fine-grained interaction perception feature representation and . After passing through a convolutional layer, asymmetric convolutional operations are respectively performed; S420: In the first branch, a horizontal convolutional kernel with a size of and a vertical convolutional kernel with a size of are sequentially used to perform convolutional operations on the features after the addition operation; in the second branch, a vertical convolutional kernel with a size of and a horizontal convolutional kernel with a size of are sequentially used to perform convolutional operations on the features after the addition operation; among them, using a horizontal convolutional kernel with a size of for convolutional operation is used to capture local feature changes in space, focusing on the local action patterns of individual behaviors in space; using a vertical convolutional kernel with a size of for convolutional operation is used to obtain global context information in the time dimension; S430: Concatenate the feature maps after the above two types of convolutional operations in the first branch and the second branch, so as to achieve deep fusion between different types of features and obtain a fused feature representation.

[0014] A group behavior recognition system based on a cross-feature interaction Transformer includes: A feature extraction module, configured to obtain a video clip, and extract features from the input video frames through appearance and pose dual-branch feature streams to obtain appearance features and pose features; A cross-feature correction module, configured to perform encoding on the appearance features and pose features, then achieve feature interaction through pooling and concatenation operations, generate and split vectors through an MLP, and finally output a cross-feature correction result through channel-level multiplication; The cross - feature interaction Transformer module is used to receive the cross - feature correction result, integrate spatial position information using positional encoding, implement dynamic feature calibration through a gated bottleneck block, and then generate queries, keys, and values through linear projection. It combines the cross - attention mechanism to generate interaction - aware features for group behavior recognition; The feature fusion module is used to receive the interaction - aware features and fuse them through asymmetric convolution to obtain the fused feature representation; The group behavior recognition module is used to input the fused feature representation into a fully - connected layer for dimensional transformation to match the number of group behavior categories. The output of the fully - connected layer is converted into a probability distribution through the Softmax layer to determine the confidence of each group behavior category, thereby completing the recognition of group behaviors.

[0015] A computer device includes a memory and a processor. The memory stores a computer program, and when the processor executes the computer program, it implements the steps of the group behavior recognition method based on cross - feature interaction Transformer.

[0016] A computer - readable storage medium stores a computer program, and when the computer program is executed by a processor, it implements the steps of the group behavior recognition method based on cross - feature interaction Transformer.

[0017] The above - mentioned group behavior recognition method and system based on cross - feature interaction Transformer aim to effectively model and utilize the complementary information between different feature streams. First, a cross - feature correction module is developed to optimize the feature representation, reduce redundant information, and enhance global information at the same time. Second, a cross - feature interaction Transformer that combines the cross - attention mechanism and the gated bottleneck block is designed to deeply analyze the complex interactions between different features. In addition, compared with the standard symmetric convolution using equal - size convolutional kernels, the present invention uses an asymmetric convolution fusion module to integrate spatial local details and temporal global context information. Finally, group behavior recognition is completed through a fully - connected layer and a softmax layer, and it can achieve highly competitive recognition performance in the group behavior recognition task. Description of the Drawings

[0018] Figure 1 It is a flowchart of the group behavior recognition method based on cross - feature interaction Transformer in an embodiment of the present invention; Figure 2 It is a schematic diagram of the overall framework principle of the group behavior recognition method based on cross - feature interaction Transformer in an embodiment of the present invention; Figure 3 It is a schematic diagram of the cross - feature correction module in an embodiment of the present invention; Figure 4 Schematic diagram of the cross - feature interaction module in an embodiment of the present invention; Figure 5 Schematic diagram of the gated bottleneck block in an embodiment of the present invention; Figure 6 Schematic diagram of the feature fusion module composed of asymmetric convolutions in an embodiment of the present invention. Detailed implementation manners

[0019] In order to enable those skilled in the art to better understand the technical solutions of the present invention, the present invention will be further described in detail below with reference to the accompanying drawings.

[0020] In one embodiment, as Figure 1 shown, a group behavior recognition method based on a cross - feature interaction Transformer, the method includes the following steps: S100: Obtain a video clip and input it into a feature extraction network, and extract features from the input video frames through appearance and pose dual - branch feature streams to obtain appearance features and pose features; S200: Input the appearance features and pose features into a cross - feature correction module. After encoding the appearance features and pose features, perform a pooling and splicing operation to achieve feature interaction, then generate and split vectors through an MLP, and finally output a cross - feature correction result through channel - level multiplication correction; S300: Input the cross - feature correction result into a cross - feature interaction Transformer module, use positional encoding to integrate spatial position information, implement dynamic feature calibration through a gated bottleneck block, then generate queries, keys, and values through linear projection, and generate interaction - aware features for group behavior recognition in combination with the cross - attention mechanism; S400: Input the interaction - aware features into a feature fusion module, and perform fusion through asymmetric convolutions to obtain a fused feature representation; S500: Input the fused feature representation into a fully - connected layer for dimensional transformation to match the number of group behavior categories, and convert the output of the fully - connected layer into a probability distribution through a Softmax layer to determine the confidence of each group behavior category, thereby completing the recognition of group behaviors.

[0021] Specifically, the core of the framework of the method of the present invention lies in adopting an innovative cross - feature interaction mechanism to make full use of the complementary information between different features, thereby improving the accuracy and robustness of group behavior recognition. Through a carefully designed modular structure, the model can capture the complex interaction relationships among the members within the group and map them to the behavior recognition task. As Figure 2 shown, the model framework is composed of multiple key components, including a feature extraction module, a cross - feature correction module, a cross - feature interaction Transformer, and a feature fusion module.

[0022] The above-mentioned group behavior recognition method based on cross-feature interaction Transformer: 1) A group behavior recognition method based on cross-feature interaction Transformer is proposed, which makes full use of the complementary information between different features. This framework integrates the cross-attention mechanism and the gated bottleneck block to enhance the information flow between features and achieve fine-grained feature interaction. 2) Design a cross-feature correction module to perform channel-level feature correction to enhance the global spatial information and screen and correct irrelevant information at the same time. 3) Construct an asymmetric convolution fusion module to effectively integrate fine-grained feature representations.

[0023] In one embodiment, S100 includes: S110: Uniformly sample frame videos , and each video frame contains actor bounding boxes, denoted as , where respectively represent the height and width of the video frame; S120: In the pose branch, given the input video frame and the individual bounding box, use HRNet as the pose estimation network to extract the two-dimensional joint coordinates of the individual, where 17 represents the number of predicted joints; this network captures the key parts of the human body, such as the head, wrists, shoulders, etc., by extracting the joints of individual actors; S130: The joint coordinates generate a pose feature vector through a fully connected layer, where represents the dimension of the pose feature vector; S140: In the appearance branch, given the input video frame and the individual bounding box, use the Inception-v3 feature extraction network as the backbone network to generate a high-dimensional feature map. Use the RoIAlign operation to crop the corresponding region on the feature map according to the bounding box coordinates, and adjust this region to a unified size. Then, compress the aligned feature map into a feature vector through average pooling operation to retain the appearance information of the individual actor. The appearance feature of the individual actor is represented as , where represents the dimension of the appearance feature vector.

[0024] Specifically, in order to effectively extract features from the input video segment , the present invention uses a two-branch feature stream to extract features from the input video frame.

[0025] Furthermore, compared with visual appearance features that are vulnerable to occlusion and illumination, pose features are more robust to illumination changes and partial occlusion, can ignore the influence of perspective changes, and can perform activity recognition without infringing on the privacy of individual actors. However, pose features may have certain limitations in capturing background environments and texture details, and it is not easy to distinguish different individual actors. In complex group scenarios, the present invention combines visual appearance features and pose features to utilize their respective advantages. Appearance features can be used to distinguish different individual actors, while pose features can be used to analyze the dynamic interactions and motion patterns between individual actors. Through this feature interaction method, the accuracy and robustness of recognition are improved.

[0026] Feature representations learned by different backbone networks may contain uncertain noise, and the features of one branch can, to a certain extent, filter and correct irrelevant information for the features of another branch. Therefore, to overcome these limitations and make full use of the advantages between different features, the present invention designs a novel Cross-Feature Rectification Module (CFRM) to enable the network to fully explore the complex interaction relationships existing between different types of features. After the parallel feature extraction branches, this module performs channel-level feature rectification, as Figure 3 shown, aiming to capture valuable complementary information between different feature branches to enhance the recognition performance of group behaviors.

[0027] In one embodiment, S200 includes: S210: After extracting pose features and appearance features , through a 1×1 convolutional layer combined with a ReLU activation function, convert the individual features into -dimensional pose feature vectors and appearance feature vectors ; through this conversion method, high-dimensional feature information is effectively compressed and encoded, providing a basis for subsequent feature processing; S220: Along the channel dimension, both and use global max pooling and global average pooling operations to retain rich global scene information. Concatenate the four resulting vector representations obtained through the above pooling operations to form a concatenated feature vector to explore the interaction between appearance features and pose features and mine meaningful complementary information; where C represents the dimension of the feature vector; S230: Use a multi-layer perceptron MLP network with a Sigmoid activation function to obtain the final vector representation from ​ and further split into and ; S240: Multiply and separately with and at the channel level to obtain the final cross - feature channel - level corrected pose feature vector and the corrected appearance feature vector .

[0028] Specifically, the input feature and generate two attention vectors and along the spatial axis through the cross - feature correction module, thereby effectively understanding and processing the complex spatial relationships between different input features. The design goal of the entire cross - feature correction module is to enhance the global spatial information by integrating the representations from different feature streams in the channel dimension, while filtering and correcting irrelevant information.

[0029] In one embodiment, S210 is specifically: , ; wherein, represents the pose feature vector after being processed by a 1×1 convolution and the ReLU activation function, represents the appearance feature vector after being transformed in the channel dimension and activated by ReLU; S230 is specifically: ; wherein, represents the concatenated feature vector, represents the multi - layer perceptron MLP network, represents the correction weight of the pose feature, represents the correction weight of the appearance feature; S240 is specifically: ; wherein, represents channel - level multiplication, performing element - by - element multiplication on the elements of each channel in the feature map, represents the corrected pose feature vector, represents the corrected appearance feature vector.

[0030] Furthermore, the present invention designs a cross - feature interaction Transformer, whose core goal is to jointly capture and process the internal connections between different features. As Figure 4As shown, the architecture diagram of the cross-feature interaction Transformer is introduced in detail, mainly including two parts: the gated bottleneck block and the cross-attention module.

[0031] In one embodiment, in S300, positional encoding is used to integrate spatial position information, dynamic feature calibration is achieved through the gated bottleneck block, and then queries, keys, and values are generated through linear projection, including: S310: Use positional encoding to integrate spatial position information and fuse it into the output representation of the cross-feature correction module; S320: The gated bottleneck block receives the and integrated by positional encoding as inputs, and respectively learn element-wise weight mappings and , thereby being able to dynamically adjust the information flow between feature streams. The calculation methods of and are as follows: ; ; Among them, represents the addition operation in the channel dimension; S330: The output representation of the gated bottleneck block is: ; Among them, represents the element-wise dot product operation; S340: After the gated bottleneck block, use learnable weight parameters to linearly project the output features of the gated bottleneck block, and respectively transform and generate queries , keys and values as well as queries , keys and values . The calculation process is expressed as: ; ; Among them, , and represent learnable weight parameters.

[0032] Specifically, since group behavior recognition not only requires understanding the behavior of individuals but also the positional relationships between the actors. Positional Encoding (PE) is first used to integrate the spatial position information of individual actors and fuse it into the output representation of the cross-feature correction module. Secondly, the feature representation after positional encoding is used as input, and dynamic feature calibration is achieved through the gated bottleneck module respectively, and then queries are generated through linear projection , keys and values as well as queries , keys and values , and then the cross-attention mechanism is used to generate the feature representation for interactive perception in group behavior recognition and , as Figure 5 shown. This design can make full use of the complementary information between different features, dynamically learn the interaction relationships between different features, rather than simply processing each feature independently. By considering the interactions between features, the model can better understand the context information, which is particularly important for group behavior recognition

[0033] In one embodiment, generating the interactive perception feature for group behavior recognition by combining the cross-attention mechanism in S300 includes: S350: Using the scaled dot-product attention mechanism, calculate the global attention map between and , and then aggregate it with the value , and calculate the global attention map between and , and then aggregate it with the value to obtain the interactive perception feature representation and , and the calculation process of its cross-attention is expressed as: ; where the softmax function is used to calculate the similarity between the query and the key , and calculate the similarity between the query and the key ; dim represents the dimension of the query vector and the key vector; in the cross-attention mechanism, the information of the appearance feature is transmitted to the pose feature to achieve comprehensive cross-feature interaction. This interaction mechanism enables the model to effectively share information between different features, thereby enhancing the richness and complementarity of the feature representation

[0034] S360: To enhance the training stability of the model, the corrected pose feature vectors after integrating the positional encoding are and the appearance feature vectors are respectively added to the pose interaction perception features and the appearance interaction perception features output by the cross-attention. Subsequently, layer normalization operations are performed to stabilize the training process. Finally, the features are further transformed through a feed-forward neural network to generate more fine-grained interaction perception feature representations and . In the information exchange stage, the model adopts two independent feature streams, and enhances the information flow between features and realizes fine-grained feature interaction through the cross-feature interaction Transformer module, thereby generating more fine-grained feature representations and .

[0035] Through the above design, the cross-feature interaction Transformer can effectively integrate the information between different features, generate feature representations with high complementarity and context awareness, thereby improving the accuracy and robustness of group behavior recognition.

[0036] Furthermore, the present invention designs an asymmetric convolution fusion module, aiming to more effectively integrate the fine-grained feature representations and to achieve accurate group behavior recognition. The asymmetric convolution fusion module adopts horizontal and vertical convolutional kernels to capture features in different directions in the image or video frame, so as to enhance the perception ability of spatial local details and temporal global context information.

[0037] In one embodiment, S400 includes: S410: Perform an addition operation on the more fine-grained interaction perception feature representations and . After passing through the convolutional layer, perform asymmetric convolution operations respectively; S420: In the first branch, sequentially adopt a horizontal convolutional kernel of size and a vertical convolutional kernel of size to perform convolution operations on the features after the addition operation; in the second branch, sequentially adopt a vertical convolutional kernel of size and a horizontal convolutional kernel of size to perform convolution operations on the features after the addition operation; wherein, using a horizontal convolutional kernel of size for convolution operation is used to capture local feature changes in space, focusing on the local action patterns of individual behaviors in space; using a vertical convolutional kernel of size for convolution operation is used to obtain global context information in the time dimension; S430: Concatenate the feature maps after the above two types of convolution operations of the first branch and the second branch, so as to achieve deep fusion between different types of features and obtain the fused feature representation.

[0038] Specifically, as Figure 6 shown, this module uses and sized convolutional kernels to aggregate inter-frame features, so as to achieve deep fusion between different types of features.

[0039] Finally, after the feature fusion module generates the comprehensive feature representation, these features are usually sent into a fully connected layer FC. The fully connected layer can convert the fused features into scores related to the group behavior categories by learning the mapping relationship between the feature representation and the categories; the output of the fully connected layer is usually sent into the Softmax layer. The role of the Softmax function is to convert the scores output by the fully connected layer into a probability distribution, representing the possibility of each group behavior category. The output of the Softmax layer is a probability vector, where each element represents the confidence of the corresponding group behavior category. By comparing these probability values, the model can determine the group behavior category that the input data most likely belongs to.

[0040] To verify the effectiveness of the proposed method, the performance of the proposed model is tested on three widely used datasets VD, CAD, and CAED, and the multi-class classification accuracy (MCA) and mean per-class classification accuracy (MPCA) are used as evaluation metrics. First, the execution details of the model (mainly including the backbone network, training strategy, hyperparameter settings, etc.) and multiple variant models are introduced in detail. Then, the comparison results of the proposed method with other previous SOTA methods in terms of recognition accuracy are reported.

[0041] The present invention selects the cross-entropy loss function to optimize the training process of the network model. In the feature extraction part, the present invention respectively uses VGG-16 and HRNet as the backbone networks. For the volleyball dataset, frames are randomly selected during the training stage, and the resolution of the input video frames is , while frames are used during the test stage, and a total of 100 rounds of training are performed. For the collective activity dataset, frames are used in both the training and test stages, and the resolution of the input video frames is , and a total of 40 rounds of training are performed. To ensure the fairness of the comparison, the maximum number of individual actors in the volleyball dataset scenario is set to , and in the collective activity dataset it is set to . For the appearance feature branch and the pose feature branch, the channel dimensions of the individual feature vectors are respectively set to and . In the appearance feature extraction branch, the RoIAlign method is used to extract the feature map of individual actors, and the cropping size is . Subsequently, the appearance features and pose features are embedded into the same dimension . For the cross-feature interaction Transformer, the present invention adopts 2 layers of encoder / decoder, each layer contains 8 attention heads, and the number of stacked cross-feature interaction Transformers is . For CAD and CAED, following the conventional settings in the prior art, the present invention combines the "crossing" class and the "walking" class into the "moving" class. All experiments of the present invention are based on the Ubuntu system, using the Pytorch deep learning framework, and are executed on 2 NVIDIA GeForce RTX 3090 GPUs. The comparison of the recognition performance of different algorithms on the volleyball dataset (VD), collective activity dataset (CAD), and collective activity extended dataset (CAED) is shown in Table 1, and the multi-class classification accuracy (MCA) and the average per-class classification accuracy (MPCA) are used as evaluation metrics. "_" indicates that "crossing" and "walking" are combined into "moving", and " / " indicates that the result is not provided.

[0042] Table 1

[0043] As shown in Table 1, the recognition performance of the method of the present invention and other comparison methods on the volleyball dataset, collective activity dataset, and collective activity extended dataset is compared. In the collective activity dataset CAD, the method proposed by the present invention exceeds all previous SOTA methods in terms of the MPCA index, reaching an accuracy of 97.0%. This remarkable recognition result proves the effectiveness of the proposed model in the group behavior recognition task.

[0044] In the volleyball dataset VD, the network architecture of the present invention performs outstandingly among the multi-feature input methods. In particular, the cross-feature interaction Transformer using VGG-16 as the backbone network in the present invention, compared with three methods with the same settings of AT, SACRF, and TCE+STBiP, in terms of the MCA index, the recognition accuracy is improved by 1.2%, 0.6%, and 0.6% respectively. The experimental results show that by introducing the cross-feature interaction mechanism, the method of the present invention can effectively enhance the discriminability of feature representation, thus achieving better recognition results in the group behavior recognition task. At the same time, the method of the present invention only uses RGB images as input, and the MCA on the volleyball dataset is 94.7%, further verifying the importance of mining complementary information between different types of features for group behavior recognition.

[0045] In the Collective Activity Expansion Dataset (CAED), the method of the present invention uses VGG-16 as the backbone network, and its recognition performance is improved by 5.9% and 0.2% compared with the SIM and MLST-Former methods respectively. In addition, to further evaluate the impact of different backbone networks on performance (while keeping all other settings the same and only changing the type of backbone network), ResNet-18 and Inception-v3 are used to replace the original backbone network VGG-16 respectively. The experimental results show that using VGG-16 and HRNet as the backbone networks achieves an accuracy improvement of 0.6% and 0.4% respectively compared with using ResNet-18 and Inception-v3 as the backbone networks in terms of the MPCA metric of the volleyball dataset. The method of the present invention uses VGG-16 and HRNet as the backbone networks to extract appearance features and pose features in complex group scenarios respectively, and achieves the best performance on the three public datasets listed in Table 1. This superior recognition performance is attributed to the model's ability to effectively mine and utilize meaningful complementary information while effectively screening out irrelevant information.

[0046] The present invention proposes a network model based on cross-feature interaction Transformer, aiming to achieve accurate group behavior recognition. To deeply explore the information interaction between different features in the group behavior recognition task, the present invention constructs a cross-feature interaction Transformer integrating cross-attention mechanism and gated bottleneck block, which can generate rich and complementary feature information. In addition, an asymmetric convolution fusion block is further designed to integrate refined feature representations. Extensive experimental results show that the method of the present invention exhibits better performance in the group behavior recognition task compared with the previous SOTA methods.

[0047] In one embodiment, a group behavior recognition system based on cross-feature interaction Transformer is also provided, including: A feature extraction module, configured to obtain video segments, extract features from the input video frames through the appearance and pose dual-branch feature streams, and obtain appearance features and pose features; A cross-feature correction module, configured to encode the appearance features and pose features, perform feature interaction through pooling and splicing operations, then generate and split vectors through MLP, and finally output the cross-feature correction result after channel-level multiplication correction; A cross-feature interaction Transformer module, configured to receive the cross-feature correction result, integrate spatial position information using position encoding, perform dynamic feature calibration through the gated bottleneck block, then generate queries, keys, and values through linear projection, and generate interaction-aware features for group behavior recognition in combination with the cross-attention mechanism; A feature fusion module, configured to receive interaction perception features, perform fusion through asymmetric convolution, and obtain a fused feature representation; A group behavior recognition module, configured to input the fused feature representation into a fully connected layer for dimensional transformation to match the number of group behavior categories, and convert the output of the fully connected layer into a probability distribution through a Softmax layer to determine the confidence of each group behavior category, thereby completing the recognition of group behaviors.

[0048] For the specific limitations of a group behavior recognition system based on cross-feature interaction Transformer, reference can be made to the limitations of a group behavior recognition method based on cross-feature interaction Transformer in the above text, which will not be elaborated here. Each module in the above group behavior recognition system based on cross-feature interaction Transformer can be implemented in whole or in part by software, hardware, and their combination. The above modules can be embedded in or independent of the processor in a computer device in the form of hardware, or stored in the memory of the computer device in the form of software, so that the processor can call and execute the operations corresponding to the above modules.

[0049] A computer device, comprising a memory and a processor, where the memory stores a computer program, and the processor implements the steps of the group behavior recognition method based on cross-feature interaction Transformer when executing the computer program.

[0050] A computer-readable storage medium, on which a computer program is stored, and the computer program implements the steps of the group behavior recognition method based on cross-feature interaction Transformer when executed by a processor.

[0051] Those of ordinary skill in the art can understand that all or part of the processes in the methods of the above embodiments can be completed by instructing relevant hardware through a computer program. The computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above methods. Among them, any reference to a memory, storage, database, or other medium used in the various embodiments provided in the present application can include at least one of non-volatile and volatile memories. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, or optical memory, etc. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM), etc.

[0052] The above has introduced in detail a group behavior recognition method and system based on cross-feature interaction Transformer provided by the present invention. Specific examples are used in this article to elaborate on the principle and implementation manner of the present invention. The description of the above embodiments is only used to help understand the core idea of the present invention. It should be noted that for those of ordinary skill in the art in this technical field, without departing from the principle of the present invention, several improvements and modifications can be made to the present invention, and these improvements and modifications also fall within the protection scope of the claims of the present invention.

Claims

1. A method for group behavior recognition based on a cross-feature interaction Transformer, characterized in that The method includes the following steps: S100: Obtain a video clip and input it into a feature extraction network. Extract features from the input video frames through an appearance and pose dual-branch feature stream to obtain appearance features and pose features; S200: Input the appearance features and pose features into a cross-feature correction module. After encoding the appearance features and pose features, perform feature interaction through a pooling and splicing operation, then generate and split vectors through an MLP, and finally output a cross-feature correction result through channel-level multiplication correction; S300: Input the cross-feature correction result into a cross-feature interaction Transformer module. Integrate spatial position information using positional encoding, implement dynamic feature calibration through a gated bottleneck block, then generate queries, keys, and values through linear projection, and generate interaction-aware features for group behavior recognition by combining the cross-attention mechanism; S400: Input the interaction-aware features into a feature fusion module and fuse them through asymmetric convolution to obtain a fused feature representation; S500: Input the fused feature representation into a fully connected layer for dimensional transformation to match the number of group behavior categories. Convert the output of the fully connected layer into a probability distribution through a Softmax layer to determine the confidence of each group behavior category, thereby completing the recognition of group behaviors.

2. The method according to claim 1, wherein S100 includes: S110: Uniformly sample from video clips video frame , each video frame contains actor bounding boxes, denoted as , where represent the height and width of the video frame respectively; S120: In the pose branch, given an input video frame and an individual bounding box, HRNet is used as the pose estimation network to extract the two-dimensional joint point coordinates of the individual , where 17 represents the number of predicted joint points; S130: The joint point coordinates generate a pose feature vector through a fully connected layer , where represents the dimension of the pose feature vector; S140: In the appearance branch, given the input video frame and the individual bounding box, the Inception-v3 feature extraction network is used as the backbone network to generate a high-dimensional feature map. The RoIAlign operation is used to crop the corresponding region on the feature map according to the bounding box coordinates, and the region is adjusted to a unified size. Then, the aligned feature map is compressed into a feature vector through average pooling operation to retain the appearance information of the individual actor. The appearance feature of the individual actor is represented as , where represents the dimension of the appearance feature vector.

3. The method according to claim 2, wherein S200 includes: S210: After extracting the pose features and appearance features , through a 1×1 convolutional layer combined with the ReLU activation function, the individual features are converted into -dimensional pose feature vectors and appearance feature vectors ; S220: Along the channel dimension, both use global max pooling and global average pooling operations, and splice the four resulting vector obtained through the above pooling operations to form a spliced feature vector , to explore the interaction between appearance features and pose features; where C represents the dimension of the feature vector; S230: Use a multi-layer perceptron (MLP) network with a Sigmoid activation function to obtain the final vector representation from and further split it into , and ; S240: Multiply with respectively with and at the channel level to obtain the final cross-feature channel-level corrected pose feature vector and the corrected appearance feature vector .

4. The method according to claim 3, wherein Specifically, S210 is: ， ; Among them, represents the pose feature vector processed by 1×1 convolution and ReLU activation function, represents the appearance feature vector after channel dimension transformation and ReLU activation; Specifically, S230 is: ; Among them, represents the concatenated feature vector, represents the multi-layer perceptron MLP network, represents the correction weight of the pose feature, represents the correction weight of the appearance feature, represents the Sigmoid activation function; Specifically, S240 is: ; Among them, represents channel-level multiplication, which performs an element-wise multiplication operation on the elements of each channel in the feature map, represents the corrected pose feature vector, represents the corrected appearance feature vector.

5. The method according to claim 4, characterized in that In S300, integrating spatial position information using positional encoding, implementing dynamic feature calibration through a gated bottleneck block, and then generating queries, keys, and values through linear projection includes: S310: Integrate spatial position information using positional encoding and fuse it into the output representation of the cross-feature correction module; S320: The gated bottleneck block receives the and after position encoding integration as inputs, and respectively learns element-wise weight mappings and , thereby being able to dynamically adjust the information flow between feature streams. The calculation methods of and are as follows: ; ; Among them, represents an addition operation in the channel dimension; The output representation of the gated bottleneck block is: ; Among them, represents an element-wise dot product operation; S340: After the gated bottleneck block, linearly project the output features of the gated bottleneck block using learnable weight parameters, and transform them respectively to generate queries , keys and values as well as queries , keys and values . The calculation process is expressed as: ; ; Among them, , and represent learnable weight parameters.

6. The method according to claim 5, wherein In S300, generating interaction-aware features for group behavior recognition by combining the cross-attention mechanism includes: S350: Calculate the global attention map between and , then aggregate it with the value , and calculate the global attention map between and , then aggregate it with the value to obtain the interaction-aware feature representations and . The calculation process of its cross-attention is expressed as: ; Among them, the softmax function is used to calculate the similarity between the query and the key , and calculate the similarity between the query and the key , where dim represents the dimension of the query vector and the key vector; S360: To enhance the training stability of the model, the corrected pose feature vector after integrating the positional encoding and the appearance feature vector are respectively added to the pose interaction perception feature and the appearance interaction perception feature output by the cross-attention, and then layer normalization operation is performed to stabilize the training process. Finally, the features are further transformed by a feed-forward neural network to generate a more fine-grained interaction perception feature representation and . and the appearance feature vector are respectively added to the pose interaction perception feature and the appearance interaction perception feature output by the cross-attention, and then layer normalization operation is performed to stabilize the training process. Finally, the features are further transformed by a feed-forward neural network to generate a more fine-grained interaction perception feature representation and . and .

7. The method according to claim 6, wherein S400 includes: S410: Represent the finer-grained interaction perception features and perform an addition operation. After passing through the convolutional layer, perform asymmetric convolutional operations respectively; S420: In the first branch, a horizontal convolutional kernel of size and a vertical convolutional kernel of size are successively used to perform a convolution operation on the features after the addition operation; in the second branch, a vertical convolutional kernel of size and a horizontal convolutional kernel of size are successively used to perform a convolution operation on the features after the addition operation; wherein, the convolution operation using the horizontal convolutional kernel of size is used to capture local feature changes in space and focus on the local action patterns of individual behaviors in space; the convolution operation using the vertical convolutional kernel of size is used to obtain global context information in the time dimension; S430: Concatenate the feature maps after the above two types of convolution operations of the first branch and the second branch to achieve deep fusion between different types of features and obtain a fused feature representation.

8. A group behavior recognition system based on a cross-feature interaction Transformer, characterized in that Includes: A feature extraction module for obtaining a video clip and extracting features from the input video frames through an appearance and pose dual-branch feature stream to obtain appearance features and pose features; A cross-feature correction module for encoding the appearance features and pose features, performing feature interaction through a pooling and splicing operation, then generating and splitting vectors through an MLP, and finally outputting a cross-feature correction result through channel-level multiplication correction; A cross-feature interaction Transformer module for receiving the cross-feature correction result, integrating spatial position information using positional encoding, implementing dynamic feature calibration through a gated bottleneck block, then generating queries, keys, and values through linear projection, and generating interaction-aware features for group behavior recognition by combining the cross-attention mechanism; A feature fusion module for receiving the interaction-aware features and fusing them through asymmetric convolution to obtain a fused feature representation; The group behavior recognition module is used to input the fused feature representation into a fully connected layer for dimensional transformation to match the number of group behavior categories, and convert the output of the fully connected layer into a probability distribution through a Softmax layer to determine the confidence of each group behavior category, thereby completing the recognition of group behaviors.

9. A computer device, comprising a memory and a processor, the memory storing a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the steps of the method according to any one of claims 1 to 7.

Images