A panoramic behavior recognition method and system based on cross-layer interaction network

Abstract

The application discloses a kind of panoramic behavior identification method and system based on cross-level interactive network, comprising: input RGB panoramic video segment, extract global scene feature map and individual local feature input social grouping module;Cosine similarity is calculated by combining position and appearance features to divide subgroup, generate multi-scale social group features;Cross-level interactive network is constructed with two encoders, a decoder and actor adaptive attention module, extract each branch feature sequence, process encoder hidden representation and decoder self-attention output and calculate cross attention;Actor adaptive attention module is processed to generate refined individual, social group and global scene activity features;These features are mapped to behavior prediction, calculate multi-task loss function and optimize model to output results.The method realizes the unity of social structure modeling, cross-level interaction enhancement and multi-task collaborative learning, improves the precision and robustness of complex scene panoramic behavior identification.

Description

Technical Field

[0001] This invention belongs to the field of panoramic behavior recognition technology, and in particular relates to a panoramic behavior recognition method and system based on a cross-level interactive network. Background Technology

[0002] For decades, human behavior research has typically been divided into three levels: individual actions, interactive behaviors, and group behaviors. Traditional recognition tasks have often focused on a single level, neglecting the interplay of multiple levels of behavior in real-world scenarios. However, unpredictable changes in the number of people, the diversity of individual actions, and the complexity of group structures in real-world panoramic scenes present new challenges to behavior recognition. To address these challenges, panoramic behavior recognition (PAR) technology has emerged. It integrates multiple sensor and camera technologies, analyzing motion trajectories and regional optical flow characteristics in video image sequences to achieve comprehensive recognition and understanding of complex behaviors. It can identify common patterns and detect abnormal behaviors, demonstrating broad application prospects.

[0003] Crowd behavior exhibits certain patterns: there is interaction between individuals, and local or overall movement follows a regular pattern. The classic Bag of Words (BoW) model, originating from text analysis, is a commonly used model for crowd behavior recognition, and most anomaly detection methods are based on it to construct feature descriptors. For example, some researchers have extended the BoW model to detect suspicious events, treating low-frequency events as anomalies, and modeling the spatiotemporal volume using a probabilistic model. If a video segment cannot match an existing descriptor, it is judged as an anomaly. Other researchers have used manifold learning frameworks for anomaly detection, using Lagrange feature maps to learn local motion structures, clustering different motion patterns, and then locating anomalies using local probabilistic models.

[0004] In addition, some scholars have introduced the concepts of ROI (Region of Interest) and Eigenvalue maps, using time integration of the dynamic system to obtain the trajectory of moving particles to locate the ROI. They then use the Jacobian matrix to linearly approximate the dynamic system and identify five simple behaviors based on matrix parameters. However, this algorithm has poor applicability when there is occlusion or inconsistent feature flow. Based on block combination features, some scholars extract trajectory segments and divide image regions to obtain trajectory-oriented histograms. They then use LDA (Latent Dirichlet Allocation) to learn regular motion patterns, finding that 8×12 partitioning has the lowest error rate because smaller region partitioning more easily reflects regional connections. Other scholars use optical flow fields and block clustering to detect crowd events. They learn motion direction and speed through clustered motion features, combine this with region segmentation to generate motion types, and then compare the motion parameters of new videos to detect various types of crowd events.

[0005] In high-density crowd scenes, target pixels are few, blurry, and severely occluded. Some scholars have used FFM (Flow Field Model) for motion pattern segmentation and anomaly detection; others have proposed algorithms that suggest individual movement is influenced by local or global forces, which are related to scene layout and guide behavior, and can help predict individual direction and track targets; still others have proposed a global interest point motion detection algorithm that uses phase angle and vector magnitude to describe flow vectors and detects motion through spectral residuals, which is effective for anomaly detection in crowded scenes.

[0006] Regarding energy-based methods, some scholars use crowd behavior energy entropy to detect anomalies, analyzing changes in scene behavior entropy and pixel-to-pixel entropy distribution to locate anomalies. They found that entropy increases significantly in scenarios such as running and gathering. However, this method relies on the feature differences between abnormal and normal dynamics, and the threshold selection is crucial for different scenarios. For moving targets, other scholars have proposed RFT (Regional Flow Trajectory) to learn the semantic regions of trajectory segments, combining LDA topic models and Markov random fields to enhance the spatiotemporal correlation of trajectories, making it suitable for estimating pedestrian movement. Some scholars construct motion pattern distributions to obtain local differences, using distribution-based HMM (Hidden Markov Model) to describe them. Others have introduced sparse representation models into crowd behavior analysis, sparsely reconstructing multi-level histograms of optical flow and judging whether behavior is normal based on reconstruction errors. While HMM can predict dynamics and detect anomalies, it requires a large amount of training data and has poor generalization ability; sparse representation models, on the other hand, judge anomalies based on reconstruction errors.

[0007] Compared to traditional human behavior recognition, Panoramic Activity Recognition (PAR) offers a wider field of view, encompasses more individual behaviors, richer interactions, and more complex group relationships, providing more comprehensive scene perception information. However, while existing research has made progress in action recognition and group behavior analysis, it still faces challenges in crowded panoramic scenes: firstly, it neglects fine-grained interactions at different behavioral levels; secondly, it has limitations in handling interactions between social groups. Previous studies have employed a hierarchical modeling approach from "individual to group" and "group to global," but these approaches fail to consider the impact of cross-level interactions, making it difficult to fully utilize contextual information. Furthermore, individual interactions form the basis of global behavior, and the social group layer serves as an intermediate representation between individual interactions and global behavior; neglecting its analysis will limit the depth of understanding of global behavior. Therefore, this invention aims to construct an end-to-end network framework to capture the interaction relationships at different behavioral levels in panoramic scenes. Summary of the Invention

[0008] To address the above technical problems, this invention provides a panoramic behavior recognition method and system based on a cross-layer interactive network.

[0009] The technical solution adopted by this invention to solve its technical problem is:

[0010] A panoramic behavior recognition method based on a cross-layered interactive network, the method comprising the following steps:

[0011] S100: Input RGB panoramic video clips, use a pre-trained dilated 3D network to extract global scene feature maps, extract local features of each individual in the video through RoIAlign, obtain the individual feature set and input it into the social grouping module;

[0012] S200: Based on individual local features, combined with location and appearance features, the social grouping module calculates cosine similarity to divide subgroups and generates multi-scale social group features.

[0013] S300: Construct a cross-layered interactive network model containing two encoders, one decoder, and an actor adaptive attention module. Extract feature sequences from individuals, social groups, and global scene branches, add learned embedding vectors and learnable positional embeddings to form encoder input tokens, generate corresponding encoder hidden representations by processing the tokens, send panoramic scene branch feature sequences to the decoder, add learned embedding vectors and learnable positional embeddings to form decoder input tokens, generate decoder self-attention outputs by processing the tokens, and further calculate cross-attention based on the two encoder hidden representations and decoder self-attention outputs.

[0014] S400: The cross-attention output of the decoder is weighted and masked by the actor adaptive attention module to generate refined individual activity features, social group activity features and global scene activity features;

[0015] S500: Refined individual activity features, social group activity features, and global scene activity features are mapped to corresponding behavior predictions through independent MLP networks. The three-layer prediction loss is calculated using binary cross-entropy loss, and the loss is weighted and summed with the regularization term to form a multi-task loss function. The model is optimized to output panoramic behavior recognition results.

[0016] Preferably, the pre-trained dilated 3D network is an I3D network pre-trained on the Kinetics dataset. Extracting global scene features includes: obtaining a global scene feature map from the last convolutional layer of the I3D network and adjusting it to a preset size.

[0017] Preferably, S200 includes:

[0018] S210: Extract the center coordinates of the bounding box of an individual. As a location feature, coordinates are mapped to D-dimensional location features using MLP;

[0019] S220: By combining D-dimensional positional features with individual appearance features through element-level summation, refined individual features are obtained;

[0020] S230: Calculate the cosine similarity between the refined individual features. For each individual, select the K individuals with the highest similarity to form a subgroup, and adjust the K value to generate multi-scale social group features.

[0021] Preferably, in S300, learning embedding vectors and learnable positional embeddings are added to form input tokens for the individual branch encoder and the social group branch encoder, and learning embedding vectors and learnable positional embeddings are added to form input tokens for the decoder, specifically as follows:

[0022] The corresponding learned embedding vector is concatenated at the beginning of the feature sequences of the individual branch, social group branch, and global scene branch. Then, it is added element-wise to the individual branch, social group branch, and global scene branch respectively to form the input tokens for the individual branch encoder, social group branch encoder, and decoder, which are as follows:

[0023] ;

[0024] ;

[0025] ;

[0026] in, , and These are the input tokens for the individual branch encoder, the social group branch encoder, and the decoder, respectively. , and These are the learned embedding vectors for the individual branch, the social group branch, and the global scene branch, respectively. , and These are the location codes for individual branches, social group branches, and global scene branches, respectively. The feature sequence extracted from individual branches; These are feature sequences extracted from social group branches; These are feature sequences extracted from the global scene branch;

[0027] In S300, the token is processed by the corresponding encoder to generate the hidden representation of the corresponding encoder, including:

[0028] The input tokens for individual branches and social group branches are processed through an encoder layer based on the Transformer architecture. The encoder layer includes a self-attention layer, residual connections, layer normalization, and a feedforward network to generate hidden representations for individual branches and social group branches.

[0029] ;

[0030] ;

[0031] In S300, the decoder processes the tokens to generate the decoder's self-attention output, specifically as follows:

[0032] ;

[0033] in, Indicates self-attention. The layer normalization function is represented. Indicates a feedforward network. , and These represent the outputs of the self-attention layers in the individual branch encoder, the social group branch encoder, and the decoder, respectively. and These represent the outputs of the individual branch encoder and the social group branch encoder, respectively.

[0034] Preferably, in S300, cross-attention is calculated in the decoder based on the encoder's hidden representation and the decoder's self-attention output, including:

[0035] Decoder self-attention output The query vector is obtained through linear mapping. Individual branch encoder hidden representation The key vector of each individual branch is obtained through linear mapping. Sum value vector Social group branch encoder hidden representation The key vectors of social group branches are obtained through linear mapping. Sum value vector ;

[0036] Based on query vector Key vectors of individual branches Sum value vector Key vectors of social group branches Sum value vector Calculate cross-attention, specifically:

[0037] ;

[0038] ;

[0039] in, For individual-global cross-attention output, For group-global cross-attention output, This is the transpose of the key vector of the individual branch. This is the transpose of the key vector of the social group branch. These are the feature dimensions for the key vector and the query vector.

[0040] Preferably, S400 includes:

[0041] S410: Receives the decoder's self-attention output and the cross-attention result from the individual to the global scene as input. Based on the concatenation of the self-attention output and the cross-attention result from the individual to the global scene, it generates initial weights involving the actors through a linear layer and a sigmoid function. Specifically:

[0042] ;

[0043] in, Indicates a connection operation. Indicates a linear layer. Represents the Sigmoid function;

[0044] S420: Apply the initial weights to the individual-to-global scene cross-attention results and the social group-to-global scene cross-attention results respectively to generate a weighted individual cross-attention representation. Weighted group cross-attention representation Specifically:

[0045] ;

[0046] S430: Based on the initial weights, individual masks and group masks are generated using a binary indicator function, specifically:

[0047] ;

[0048] Among them, the binary indicator function Used to generate a mask and Output 0 and 1 respectively based on given conditions, threshold Used as a hyperparameter during training;

[0049] S440: Perform element-wise masking on the weighted individual cross-attention representation using individual masks, and perform element-wise masking on the weighted group cross-attention representation using group masks to obtain the masked individual features. and the group characteristics after masking The masked individual features and the masked group features are weighted and summed to generate a fused actor adaptive attention feature representation. Specifically:

[0050] ;

[0051] in, This indicates element-wise multiplication;

[0052] S450: The fused feature representations are processed sequentially using a feedforward network and layer normalization operations, ultimately outputting refined individual activity features. Characteristics of social group activities and global scene activity features .

[0053] Preferably, S500 includes:

[0054] S510: For individual behavioral feature representations, social group behavioral feature representations, and global scene behavioral features, corresponding readout functions are implemented through independent multilayer perceptron neural networks. , and These are used to map features to the corresponding behavior category space to generate individual behavior predictions. Social group behavior prediction and global scene behavior prediction Specifically, before generating individual behavior predictions and social group behavior predictions, the global scene feature representation is integrated into the individual behavior feature representation and social group behavior representation to optimize activity category prediction.

[0055] ;

[0056] in, For individual activity characteristics, Characteristics of social group activities For global scene activity features;

[0057] S520: The binary cross-entropy loss function is used to calculate the individual behavior prediction loss, social group behavior prediction loss, and global scene behavior prediction loss respectively. The three-layer behavior prediction losses are then weighted and summed with the regularization term to form a multi-level, multi-task loss function.

[0058] ;

[0059] in, This is the binary cross-entropy loss function, used to measure the difference between the predicted result and the true label. , , , Let represent the individual behavior prediction loss, the social group behavior prediction loss, the global scene behavior prediction loss, and the regularization term, respectively. The regularization term helps prevent the model from overfitting, thereby improving the model's generalization ability. , , These represent real labels at different behavioral levels: individual, social group, and global scenario. , , and These are the weight coefficients for individual behavior, social group behavior, global scene behavior, and regularization term, used to balance the loss contribution between different tasks.

[0060] A panoramic behavior recognition system based on a cross-layered interactive network includes:

[0061] The feature extraction module is used to input RGB panoramic video clips, extract global scene feature maps using a pre-trained dilated 3D network, extract local features of each individual in the video using RoIAlign, obtain individual feature sets, and input them into the social grouping module.

[0062] The social grouping module is used to calculate cosine similarity based on individual local features, combined with location and appearance features, to divide subgroups and generate multi-scale social group features.

[0063] The cross-level interaction module, comprising a cross-level interaction network with two encoders, one decoder, and an actor adaptive attention module, is used to extract feature sequences from individuals, social groups, and global scene branches, add learned embedding vectors and learnable positional embeddings to form encoder input tokens, generate corresponding encoder hidden representations by processing the tokens, send panoramic scene branch feature sequences to the decoder, add learned embedding vectors and learnable positional embeddings to form decoder input tokens, generate decoder self-attention outputs by processing the tokens, and further calculate cross-attention based on the two encoder hidden representations and decoder self-attention outputs.

[0064] The actor-adaptive attention module is used to perform weighted masking on the cross-attention output to generate refined individual activity features, social group activity features, and global scene activity features;

[0065] The multi-task optimization module maps refined individual activity features, social group activity features, and global scene activity features to corresponding behavior predictions through independent MLP networks. It uses binary cross-entropy loss to calculate the three-layer prediction loss and weights it with the regularization term to form a multi-task loss function, thus optimizing the model to output panoramic behavior recognition results.

[0066] A computer device includes a memory and a processor, the memory storing a computer program, the processor executing the computer program to implement the steps of a panoramic behavior recognition method based on a cross-layer interactive network.

[0067] A computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of a panoramic behavior recognition method based on a cross-layered interactive network.

[0068] The aforementioned panoramic behavior recognition method and system based on cross-layered interactive networks establishes an encoder to accurately extract features of different behavioral granularities from individuals, social groups, and the global scene, avoiding information interference between layers. A decoder is used to fuse the encoder's hidden representation and the decoder's self-attention output to calculate cross-attention, combined with the weighted masking mechanism of the actor's adaptive attention module, adaptively focusing on the cross-layered association information most relevant to the current actor. Finally, an independent MLP network is used to collaboratively predict the refined three-layered features, and multi-task loss is used for joint optimization, forcing the model to fully explore the complementarity of individual actions, group interactions, and scene semantics. This significantly improves feature discriminativeness, behavior association modeling ability, and cross-layered information fusion efficiency, ultimately achieving a comprehensive and accurate understanding from micro-level individual behavior to macro-level group scenes, greatly enhancing the recognition accuracy and robustness of complex panoramic behaviors. Attached Figure Description

[0069] Figure 1 This is a flowchart of a panoramic behavior recognition method based on a cross-layer interactive network in one embodiment of the present invention;

[0070] Figure 2 This is a schematic diagram illustrating the principle of a panoramic behavior recognition method based on a cross-layer interactive network in one embodiment of the present invention.

[0071] Figure 3 This is a schematic diagram illustrating the principle of the actor-adaptive attention module in one embodiment of the present invention. Detailed Implementation

[0072] To enable those skilled in the art to better understand the technical solution of the present invention, the present invention will be further described in detail below with reference to the accompanying drawings.

[0073] In one embodiment, such as Figure 1 and Figure 2 As shown, a panoramic behavior recognition method based on a cross-layered interactive network is proposed, the method comprising the following steps:

[0074] S100: Input RGB panoramic video clips, use a pre-trained dilated 3D network to extract global scene feature maps, and use RoIAlign (Region of Interest Align) to extract local features of each individual in the video, obtain the individual feature set and input it into the social grouping module;

[0075] S200: Based on individual local features, combined with location and appearance features, the social grouping module calculates cosine similarity to divide subgroups and generates multi-scale social group features.

[0076] S300: Construct a cross-layered interactive network model containing two encoders, one decoder, and an actor adaptive attention module. Extract feature sequences from individuals, social groups, and global scene branches, add learned embedding vectors and learnable positional embeddings to form encoder input tokens, generate corresponding encoder hidden representations by processing the tokens, send panoramic scene branch feature sequences to the decoder, add learned embedding vectors and learnable positional embeddings to form decoder input tokens, generate decoder self-attention outputs by processing the tokens, and further calculate cross-attention based on the two encoder hidden representations and decoder self-attention outputs.

[0077] S400: The cross-attention output of the decoder is weighted and masked by the actor adaptive attention module to generate refined individual activity features, social group activity features and global scene activity features;

[0078] S500: Refined individual activity features, social group activity features, and global scene activity features are mapped to corresponding behavior predictions through independent MLP (Multi-Layer Perceptron) networks. The three-layer prediction loss is calculated using binary cross-entropy loss, and the loss function is formed by weighted summation with regularization terms. The model is optimized to output panoramic behavior recognition results.

[0079] The aforementioned panoramic behavior recognition method based on a cross-layer interaction network constructs a social grouping module to analyze group dynamics, providing a structural foundation for panoramic recognition. It designs a cross-layer interaction network (CHINet) and its actor-adaptive attention mechanism, mining intra-layer / inter-layer interaction relationships between individuals / groups / the global context through independent encoder and decoder branches, and focusing on key features. Combining a multi-task loss function and a global feature fusion strategy, it collaboratively optimizes the three-layer behavior prediction. This method achieves a unified approach to social structure modeling, cross-layer interaction enhancement, and multi-task collaborative learning, significantly improving the accuracy and robustness of panoramic behavior recognition in complex scenes.

[0080] In one embodiment, the pre-trained dilated 3D network is an I3D network pre-trained on the Kinetics dataset. Extracting global scene features includes: obtaining a global scene feature map from the last convolutional layer of the I3D network and adjusting it to a preset size.

[0081] Specifically, an inflated 3D network (I3D) pre-trained on the Kinetics dataset is used as the backbone network, with RGB panoramic video clips as input. Given a resolution of... and From a group Extracting feature vectors from frame segments The global scene feature map is obtained from the last convolutional layer and resized. Next, local features for each individual are extracted using RoIAlign and resized to the same size to obtain individual features. Social grouping (SGM) is used to segment and analyze the characteristics of active social groups. ( It is the number of blocks. (It is a dimension of features), and models the interaction relationships at different behavioral levels in the interaction network stage.

[0082] In one embodiment, S200 includes:

[0083] S210: Individual location feature is the center coordinates of the bounding box. The coordinates are mapped to D-dimensional position features through MLP;

[0084] S220: By combining positional features with appearance features through element-level summation, refined individual features are obtained;

[0085] S230: Calculate the cosine similarity between the refined individual features. For each individual, select the K individuals with the highest similarity to form a subgroup, and adjust the K value to generate multi-scale social group features.

[0086] Specifically, panoramic behavior recognition encompasses individual behavior, social group behavior, and global behavior. Group behavior features are closely related to global behavior and can serve as a bridge connecting individual and global features. Previous group segmentation methods, such as HRN (Ibrahim and Mori 2018), rely on spatial proximity to divide subgroups, ignoring potential correlations in individual behavioral features, resulting in inaccuracies. This invention designs a Social Grouping Module (SGM) to segment subgroups in a more refined manner, enabling the model to more accurately capture the behavioral features of each subset within a group, thereby facilitating the modeling of complex global behavioral dynamics. This is combined with individual location features. ,in and These are the center coordinates of the bounding box. The location information is mapped to... using a multilayer perceptron (MLP). The individual features are obtained by adding the appearance features to the vector features. Then, the cosine similarity between each individual feature and the features of other individuals is calculated, and the top K individuals with the highest similarity form a subgroup. Since the features of the same individual may differ in different frames, this invention constructs social groups at different scales by adjusting the value of K, thereby enriching the subgroup information at different scales and generating more expressive group behavior features.

[0087] Furthermore, panoramic behavior recognition exhibits hierarchical characteristics, where global scene activities can be decomposed into three levels: individual behavior, group activity, and global behavior. Previous methods only considered interactions between individuals, lacking exploration of interactions within and between other levels. This invention proposes a cross-level interaction network (CHINet), whose framework consists of two encoders and one decoder, equipped with an actor adaptive attention module (A3M) to perform the decoding process. The encoder layer employs a self-attention mechanism to encode individual-level and group-level features, generating latent representations and filtering out information irrelevant to activity recognition. The decoder layer aggregates cross-level interaction information through a cross-attention mechanism, further extracting more representative scene context information to achieve cross-level information fusion.

[0088] In one embodiment, S300 adds learned embedding vectors and learnable positional embeddings to form input tokens for the individual branch encoder and the social group branch encoder, and adds learned embedding vectors and learnable positional embeddings to form decoder input tokens, specifically as follows:

[0089] The corresponding learned embedding vector is concatenated at the beginning of the feature sequences of the individual branch, social group branch, and global scene branch. Then, it is added element-wise to the individual branch, social group branch, and global scene branch respectively to form the input tokens for the individual branch encoder, social group branch encoder, and decoder, which are as follows:

[0090] ;

[0091] ;

[0092] ;

[0093] in, , and These are the input tokens for the individual branch encoder, the social group branch encoder, and the decoder, respectively. , and These are the learned embedding vectors for the individual branch, the social group branch, and the global scene branch, respectively. , and These are the location codes for individual branches, social group branches, and global scene branches, respectively. The feature sequence extracted from individual branches; These are feature sequences extracted from social group branches; These are feature sequences extracted from the global scene branch;

[0094] In S300, the token is processed by the corresponding encoder to generate the hidden representation of the corresponding encoder, including:

[0095] The input tokens for individual branches and social group branches are processed through an encoder layer based on the Transformer architecture. The encoder layer includes a self-attention layer, residual connections, layer normalization, and a feedforward network to generate hidden representations for individual branches and social group branches.

[0096] ;

[0097] ;

[0098] In S300, the decoder processes the tokens to generate the decoder's self-attention output, specifically as follows:

[0099] ;

[0100] in, Indicates self-attention. The layer normalization function is represented. Indicates a feedforward network. , and These represent the outputs of the self-attention layers in the individual branch encoder, the social group branch encoder, and the decoder, respectively. and These represent the outputs of the individual branch encoder and the social group branch encoder, respectively.

[0101] Specifically, the encoder layer of this invention adopts a Transformer encoder architecture, focusing on feature representation that contributes significantly to activity recognition.

[0102] In one embodiment, S300 calculates cross-attention in the decoder based on the encoder's hidden representation and the decoder's self-attention output, including:

[0103] Decoder self-attention output The query vector is obtained through linear mapping. Individual branch encoder hidden representation The key vector of each individual branch is obtained through linear mapping. Sum value vector Social group branch encoder hidden representation The key vectors of social group branches are obtained through linear mapping. Sum value vector ;

[0104] Based on query vector Key vectors of individual branches Sum value vector Key vectors of social group branches Sum value vector Calculate cross-attention, specifically:

[0105] ;

[0106] ;

[0107] in, For individual-global cross-attention output, For group-global cross-attention output, This is the transpose of the key vector of the individual branch. This is the transpose of the key vector of the social group branch. These are the feature dimensions for the key vector and the query vector.

[0108] Specifically, in obtaining the hidden representation of the encoder and and the decoder's self-attention output Afterwards, based on and Calculate cross-attention. The calculation method for cross-attention is similar to that for self-attention, where... and From the encoder, and From the decoder.

[0109] The output of the cross-attention decoding layer is then fed into the actor adaptive attention module (A3M) to explore the interaction between different layers (i.e., individual to global scene and social group to global scene).

[0110] In one embodiment, such as Figure 3 As shown, S400 includes:

[0111] S410: Receives the decoder's self-attention output and the cross-attention result from the individual to the global scene as input. Based on the concatenation of the self-attention output and the cross-attention result from the individual to the global scene, it generates initial weights involving the actors through a linear layer and a sigmoid function. Specifically:

[0112] ;

[0113] in, Indicates a connection operation. Indicates a linear layer. This represents the Sigmoid function; furthermore, the design motivation is to enable the model to understand the contextual information of the actor in both self-attention and cross-attention paradigms, thereby further capturing the dynamics of the actor that contribute significantly to improving recognition accuracy.

[0114] S420: To deeply analyze the key roles played by actors in complex social interactions and dynamic scene contexts, initial weights are applied to the individual-to-global scene cross-attention results and the social group-to-global scene cross-attention results, respectively, to generate a weighted individual cross-attention representation. Weighted group cross-attention representation Specifically:

[0115] ;

[0116] Furthermore, in order to eliminate irrelevant information and normalize the two input sources in the panoramic behavior recognition task, the initial weights are... Convert to two dedicated masks and Through this dual masking mechanism, the present invention can dynamically filter and integrate information from both individual and group levels.

[0117] S430: Based on the initial weights, individual masks and group masks are generated using a binary indicator function, specifically:

[0118] ;

[0119] Among them, the binary indicator function Used to generate a mask and Output 0 and 1 respectively based on given conditions, threshold Used as a hyperparameter during training;

[0120] S440: Perform element-wise masking on the weighted individual cross-attention representation using individual masks, and perform element-wise masking on the weighted group cross-attention representation using group masks to obtain the masked individual features. and masked group characteristics The masked individual features and the masked group features are weighted and summed to generate a fused actor adaptive attention feature representation. Specifically:

[0121] ;

[0122] in, This indicates element-wise multiplication;

[0123] S450: The fused feature representations are processed sequentially using a feedforward network and layer normalization operations, ultimately outputting refined individual activity features. Characteristics of social group activities and global scene activity features .

[0124] Specifically, for the self-attention module in the overall network structure, this invention stacks two Transformer encoder layers, each containing four attention heads and 512 channels. In the proposed CHINet, this invention further stacks one self-attention layer and two cross-attention layers, totaling three Transformer decoder layers, each containing six attention heads and 512 channels. To integrate the results of cross-attention, the actor-adaptive attention module is equipped with a weighting and masking mechanism, which can be used to enhance features related to specific behaviors while suppressing irrelevant information, thereby effectively mining actor information that contributes significantly to the overall behavior.

[0125] Furthermore, the multi-level multi-task loss function aims to enhance collaborative learning among different activity recognition tasks (i.e., individual actions, group activities, and global scene behavior). This invention employs a feature fusion strategy, which integrates global features into individual and social group feature representations by designing a multi-level multi-task loss function to optimize the prediction of activity categories.

[0126] In one embodiment, S500 includes:

[0127] S510: For individual behavioral feature representation, social group behavioral feature representation, and global scene behavioral feature representation, the corresponding readout functions are implemented through independent multilayer perceptron neural networks. , and These are used to map features to the corresponding behavior category space to generate individual behavior predictions. Social group behavior prediction and global scene behavior prediction Specifically, before generating individual behavior predictions and social group behavior predictions, the global scene feature representation is integrated into the individual behavior feature representation and social group behavior representation to optimize activity category prediction.

[0128] ;

[0129] in, For individual activity characteristics, Characteristics of social group activities For global scene activity features;

[0130] S520: The binary cross-entropy loss function is used to calculate the individual behavior prediction loss, social group behavior prediction loss, and global scene behavior prediction loss respectively. The three-layer behavior prediction losses are then weighted and summed with the regularization term to form a multi-level, multi-task loss function.

[0131] ;

[0132] in, This is the binary cross-entropy loss function, used to measure the difference between the predicted result and the true label. , , , Let represent the individual behavior prediction loss, the social group behavior prediction loss, the global scene behavior prediction loss, and the regularization term, respectively. The regularization term helps prevent the model from overfitting, thereby improving the model's generalization ability. , , These represent real labels at different behavioral levels: individual, social group, and global scenario. , , and These are the weight coefficients for individual behavior, social group behavior, global scene behavior, and regularization term, used to balance the loss contribution between different tasks;

[0133] Specifically, in this embodiment, the number of sample frames is set to Predefined threshold Set to 0.5, and the weighting coefficient ratio is set to... .

[0134] During the experiments, the proposed CHINet was tested on the JRDB-PAR dataset. The JRDB-PAR dataset is a benchmark dataset designed for the Panoramic Activity Recognition (PAR) task. It contains 360-degree RGB video recordings of crowded, multi-person scenes such as campuses, cafeterias, and classrooms, captured by a mobile robot. The dataset provides annotated human bounding boxes and their unique identifiers (IDs), individual behaviors, social group detection, and manually annotated social activities and global behaviors. The dataset contains 27 videos, 20 for training and 7 for testing, totaling 27,920 frames and over 628,000 human bounding boxes. It categorizes behaviors into 27 individual behavior labels, 11 social group activity labels, and 7 global behavior labels. The JRDB-PAR dataset provides researchers with a rich foundation for in-depth research on multi-level activity recognition tasks, covering individual behavior recognition, social group activity recognition, and global (panoramic) activity recognition.

[0135] Following the approach adopted by Han et al. 2022b, a comprehensive evaluation metric system is provided for the JRDB-PAR dataset to fully evaluate the performance of the proposed model across different activity levels. Individual behavior recognition is evaluated by calculating the degree of matching between each detected individual behavior and its true label, using accuracy ( ), recall rate ( )and Fraction( Score, ) was used as the evaluation indicator. A similar evaluation system was also used for social group activities and overall behavior, respectively using ) , , and , , The overall evaluation metrics for panoramic behavior recognition are three. The average of the fractions is expressed as .

[0136] In this invention, 1439 keyframes were used for model training, and another 411 keyframes were used for testing. The default resolution of these frames was 480×3760. Following the specifications of (Han et al. 2022b), this invention employs three fully connected networks (FC networks) as the readout function, and two fully connected networks to calculate the multi-task loss function. The number of sampling frames was set to... Predefined threshold The value is set to 0.5. For the self-attention module, this invention stacks two Transformer encoder layers, each containing four attention heads and 512 channels; in the proposed CHINet, this invention further stacks three Transformer decoder layers, each containing six attention heads and 512 channels. This invention chooses the cross-entropy loss function to guide the optimization process, and the weight coefficient ratio is set to... In the experimental implementation, this invention uses the Adam optimizer with fixed hyperparameters. , , The initial learning rate is set to The weight decay parameter is set to Batch size set to All experiments were conducted using the PyTorch framework on two GTX 3090 GPUs.

[0137] Panoramic Activity Recognition (PAR), involving activity recognition at three different levels, is an emerging and highly challenging task, thus lacking direct comparative methods. To ensure the fairness and validity of the comparison, this invention makes necessary adjustments and optimizations to selected comparison methods (e.g., ARG (Wu et al. 2019), SA-GAT (Ehsanpouret et al. 2020), and JRDB-Base (Martin-Martin et al. 2021)) to fit the experimental setup and evaluation protocol of this invention. Furthermore, JRDB-PAR (Han et al. 2022b) is the baseline method for the JRDB-PAR dataset. This invention compares and analyzes the proposed CHINet with previous state-of-the-art methods on the JRDB-PAR dataset.

[0138] Experimental data demonstrates that the proposed model outperforms existing comparative methods in overall performance. Specifically, CHINet achieves significant improvements in social group activity recognition, reaching [specific performance metrics]. of (Accuracy) of (Recall rate) and of Compared to JRDB-PAR (Hanet al. 2022b), a novel hierarchical graph neural network, this invention significantly improves these metrics. , and This fully demonstrates the effectiveness of the present invention in mining social group representations.

[0139] The above-mentioned panoramic behavior recognition method based on cross-layer interactive networks has the following beneficial effects:

[0140] (1) Construct a social grouping module, focusing on the semantic information of individual behavior, to analyze the social structure and group activities in the panoramic scene, so as to capture the complex social dynamics within and between groups, and to provide complementary information between the individual level and the global level, so as to provide a structural representation basis for panoramic behavior recognition.

[0141] (2) Design a cross-layer interactive network CHINet, a novel interactive reasoning framework consisting of two encoders and one decoder, to deeply explore the intra-layer and inter-layer interactive relationships between different behavioral activity levels. At the same time, an adaptive attention module for interactive actors is introduced to enhance features related to the target behavior, suppress interference from irrelevant information, and improve the model's ability to represent complex scenes.

[0142] (3) A multi-level, multi-task loss function was designed, and a feature fusion strategy was adopted to integrate global features into individual and social group feature representations in order to optimize the prediction of activity categories and promote collaborative learning among different behavior recognition tasks.

[0143] In one embodiment, a panoramic behavior recognition system based on a cross-layered interactive network is also provided, comprising:

[0144] The feature extraction module is used to input RGB panoramic video clips, extract global scene feature maps using a pre-trained dilated 3D network, extract local features of each individual in the video using RoIAlign, obtain individual feature sets, and input them into the social grouping module.

[0145] The social grouping module is used to calculate cosine similarity based on individual local features, combined with location and appearance features, to divide subgroups and generate multi-scale social group features.

[0146] The cross-level interaction module, comprising a cross-level interaction network with two encoders, one decoder, and an actor adaptive attention module, is used to extract feature sequences from individuals, social groups, and global scene branches, add learned embedding vectors and learnable positional embeddings to form encoder input tokens, generate corresponding encoder hidden representations by processing the tokens, send panoramic scene branch feature sequences to the decoder, add learned embedding vectors and learnable positional embeddings to form decoder input tokens, generate decoder self-attention outputs by processing the tokens, and further calculate cross-attention based on the two encoder hidden representations and decoder self-attention outputs.

[0147] The actor-adaptive attention module is used to perform weighted masking on the cross-attention output to generate refined individual activity features, social group activity features, and global scene activity features;

[0148] The multi-task optimization module maps refined individual activity features, social group activity features, and global scene activity features to corresponding behavior predictions through independent MLP networks. It uses binary cross-entropy loss to calculate the three-layer prediction loss and weights it with the regularization term to form a multi-task loss function, thus optimizing the model to output panoramic behavior recognition results.

[0149] For specific limitations regarding the panoramic behavior recognition system based on a cross-layer interactive network, please refer to the limitations of the panoramic behavior recognition method based on a cross-layer interactive network mentioned above, which will not be repeated here. Each module in the aforementioned panoramic behavior recognition system based on a cross-layer interactive network can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device in hardware form, or stored in the memory of a computer device in software form, so that the processor can call and execute the corresponding operations of each module.

[0150] A computer device includes a memory and a processor, the memory storing a computer program, the processor executing the computer program to implement the steps of a panoramic behavior recognition method based on a cross-layer interactive network.

[0151] A computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of a panoramic behavior recognition method based on a cross-layered interactive network.

[0152] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the methods described above. Any references to memory, storage, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, or optical storage, etc. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM), etc.

[0153] The above provides a detailed description of the panoramic behavior recognition method and system based on a cross-layer interactive network provided by this invention. Specific examples have been used to illustrate the principles and implementation methods of this invention. The descriptions of the embodiments above are merely for the purpose of helping to understand the core ideas of this invention. It should be noted that those skilled in the art can make various improvements and modifications to this invention without departing from its principles, and these improvements and modifications also fall within the protection scope of the claims of this invention.

Claims

1. A panoramic behavior recognition method based on a cross-layered interactive network, characterized in that, The method includes the following steps: S100: Input RGB panoramic video clips, use a pre-trained dilated 3D network to extract global scene feature maps, extract local features of each individual in the video through RoIAlign, obtain the individual feature set and input it into the social grouping module; S200: Based on individual local features, combined with location and appearance features, the social grouping module calculates cosine similarity to divide subgroups and generates multi-scale social group features. S300: Construct a cross-layered interactive network model containing two encoders, one decoder, and an actor adaptive attention module. Extract feature sequences from individuals, social groups, and global scene branches, add learned embedding vectors and learnable positional embeddings to form encoder input tokens, generate corresponding encoder hidden representations by processing the tokens, send panoramic scene branch feature sequences to the decoder, add learned embedding vectors and learnable positional embeddings to form decoder input tokens, generate decoder self-attention outputs by processing the tokens, and further calculate cross-attention based on the two encoder hidden representations and decoder self-attention outputs. S400: The cross-attention output of the decoder is weighted and masked by the actor adaptive attention module to generate refined individual activity features, social group activity features and global scene activity features; S500: Refined individual activity features, social group activity features, and global scene activity features are mapped to corresponding behavior predictions through independent MLP networks. The three-layer prediction loss is calculated using binary cross-entropy loss, and the loss is weighted and summed with the regularization term to form a multi-task loss function. The model is optimized to output panoramic behavior recognition results.

2. The method according to claim 1, characterized in that, The pre-trained dilated 3D network is an I3D network pre-trained on the Kinetics dataset. Extracting global scene features includes: obtaining the global scene feature map from the last convolutional layer of the I3D network and adjusting it to a preset size.

3. The method according to claim 2, characterized in that, S200 includes: S210: Extract the center coordinates of the bounding box of an individual. As a location feature, coordinates are mapped to D-dimensional location features using MLP; S220: By combining D-dimensional positional features with individual appearance features through element-level summation, refined individual features are obtained; S230: Calculate the cosine similarity between the refined individual features. For each individual, select the K individuals with the highest similarity to form a subgroup, and adjust the K value to generate multi-scale social group features.

4. The method according to claim 3, characterized in that, In S300, learned embedding vectors and learnable positional embeddings are added to form input tokens for the individual branch encoder and the social group branch encoder, and learned embedding vectors and learnable positional embeddings are added to form input tokens for the decoder, specifically as follows: The corresponding learned embedding vector is concatenated at the beginning of the feature sequences of the individual branch, social group branch, and global scene branch. Then, it is added element-wise to the individual branch, social group branch, and global scene branch respectively to form the input tokens for the individual branch encoder, social group branch encoder, and decoder, which are as follows: ； ； ； in, , and These are the input tokens for the individual branch encoder, the social group branch encoder, and the decoder, respectively. , and These are the learned embedding vectors for the individual branch, the social group branch, and the global scene branch, respectively. , and These are the location codes for individual branches, social group branches, and global scene branches, respectively. The feature sequence extracted from individual branches; These are feature sequences extracted from social group branches; These are feature sequences extracted from the global scene branch; In S300, the token is processed by the corresponding encoder to generate the hidden representation of the corresponding encoder, including: The input tokens for individual branches and social group branches are processed through an encoder layer based on the Transformer architecture. The encoder layer includes a self-attention layer, residual connections, layer normalization, and a feedforward network to generate hidden representations for individual branches and social group branches. ； ； In S300, the decoder processes the tokens to generate the decoder's self-attention output, specifically as follows: ； in, Indicates self-attention. The layer normalization function is represented. Indicates a feedforward network. , and These represent the outputs of the self-attention layers in the individual branch encoder, the social group branch encoder, and the decoder, respectively. and These represent the outputs of the individual branch encoder and the social group branch encoder, respectively.

5. The method according to claim 4, characterized in that, In S300, cross-attention is calculated in the decoder based on the encoder's hidden representation and the decoder's self-attention output, including: Decoder self-attention output The query vector is obtained through linear mapping. Individual branch encoder hidden representation The key vector of each individual branch is obtained through linear mapping. Sum value vector Social group branch encoder hidden representation The key vectors of social group branches are obtained through linear mapping. Sum value vector ; Based on query vector Key vectors of individual branches Sum value vector Key vectors of social group branches Sum value vector Calculate cross-attention, specifically: ； in, For individual-global cross-attention output, For group-global cross-attention output, This is the transpose of the key vector of the individual branch. This is the transpose of the key vector of the social group branch. These are the feature dimensions for the key vector and query vector.

6. The method according to claim 5, characterized in that, The S400 includes: S410: Receives the decoder's self-attention output and the cross-attention result from the individual to the global scene as input. Based on the concatenation of the self-attention output and the cross-attention result from the individual to the global scene, it generates initial weights involving the actors through a linear layer and a sigmoid function. Specifically: ； in, Indicates a connection operation. Indicates a linear layer. Represents the Sigmoid function; S420: Apply the initial weights to the individual-to-global scene cross-attention results and the social group-to-global scene cross-attention results respectively to generate a weighted individual cross-attention representation. Weighted group cross-attention representation Specifically: ； S430: Based on the initial weights, individual masks and group masks are generated using a binary indicator function, specifically: ； Among them, the binary indicator function Used to generate a mask and Output 0 and 1 respectively based on given conditions, threshold Used as a hyperparameter during training; S440: Perform element-wise masking on the weighted individual cross-attention representation using individual masks, and perform element-wise masking on the weighted group cross-attention representation using group masks to obtain the masked individual features. and masked group characteristics The masked individual features and the masked group features are weighted and summed to generate a fused actor adaptive attention feature representation. Specifically: ； in, This indicates element-wise multiplication; S450: The fused feature representations are processed sequentially using a feedforward network and layer normalization operations, ultimately outputting refined individual activity features. Characteristics of social group activities and global scene activity features .

7. The method according to claim 6, characterized in that, The S500 includes: S510: For individual behavioral feature representations, social group behavioral feature representations, and global scene behavioral features, corresponding readout functions are implemented through independent multilayer perceptron neural networks. , and These are used to map features to the corresponding behavior category space to generate individual behavior predictions. Social group behavior prediction and global scene behavior prediction Specifically, before generating individual behavior predictions and social group behavior predictions, the global scene feature representation is integrated into the individual behavior feature representation and social group behavior representation to optimize activity category prediction. ； in, For individual activity characteristics, Characteristics of social group activities For global scene activity features; S520: The binary cross-entropy loss function is used to calculate the individual behavior prediction loss, social group behavior prediction loss, and global scene behavior prediction loss respectively. The three-layer behavior prediction losses are then weighted and summed with the regularization term to form a multi-level, multi-task loss function. ； in, This is the binary cross-entropy loss function, used to measure the difference between the predicted result and the true label. , , , Let represent the individual behavior prediction loss, the social group behavior prediction loss, the global scene behavior prediction loss, and the regularization term, respectively. The regularization term helps prevent the model from overfitting, thereby improving the model's generalization ability. , , These represent real labels at different behavioral levels: individual, social group, and global scenario. , , and These are the weight coefficients for individual behavior, social group behavior, global scene behavior, and regularization term, used to balance the loss contribution between different tasks.

8. A panoramic behavior recognition system based on a cross-layered interactive network, characterized in that, include: The feature extraction module is used to input RGB panoramic video clips, extract global scene feature maps using a pre-trained dilated 3D network, extract local features of each individual in the video using RoIAlign, obtain individual feature sets, and input them into the social grouping module. The social grouping module is used to calculate cosine similarity based on individual local features, combined with location and appearance features, to divide subgroups and generate multi-scale social group features. The cross-level interaction module, comprising a cross-level interaction network with two encoders, one decoder, and an actor adaptive attention module, is used to extract feature sequences from individuals, social groups, and global scene branches, add learned embedding vectors and learnable positional embeddings to form encoder input tokens, generate corresponding encoder hidden representations by processing the tokens, send panoramic scene branch feature sequences to the decoder, add learned embedding vectors and learnable positional embeddings to form decoder input tokens, generate decoder self-attention outputs by processing the tokens, and further calculate cross-attention based on the two encoder hidden representations and decoder self-attention outputs. The actor-adaptive attention module is used to perform weighted masking on the cross-attention output to generate refined individual activity features, social group activity features, and global scene activity features; The multi-task optimization module maps refined individual activity features, social group activity features, and global scene activity features to corresponding behavior predictions through independent MLP networks. It uses binary cross-entropy loss to calculate the three-layer prediction loss and weights it with the regularization term to form a multi-task loss function, thus optimizing the model to output panoramic behavior recognition results.

9. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 7.

Images