Double-path fusion group behavior recognition method based on comparative learning
Through the dual-path fusion method of comparative learning, combined with time and space encoder, the comparative learning loss function is designed, which solves the problem of neglecting space-time attention and in-depth fusion of dual-paths in the existing technology, and realizes the complex space-time interaction relationship of individuals in the video modeling and group behavior recognition effect.
Patent Information
- Application Number
- CN202411007132.7
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-25
- Publication Date
- 2024-11-08
AI Technical Summary
The existing group behavior recognition methods ignore the joint consideration of space-time attention, and do not deeply explore different fusion methods in terms of dual-path fusion, resulting in the order of temporal and spatial interactions between individuals that do not adapt to the diversity of group behaviors, and ignore individuals. Class-level interaction.
Using a dual-path fusion method based on contrast learning, video features are extracted using Inception-v3, time-space and space-time interaction relationships are modeled through time encoder and spatial encoder, class feature interaction is enhanced in combination with multi-head attention mechanism, and designed Comparative learning loss function maintains individual characteristics consistency and increases category differences.
Effectively model the complex spatiotemporal and spatial interactions between individuals in videos, reduce individual category confusion, improve group behavior recognition effect, and enhance feature discrimination ability through complementary spatiotemporal and spatial context information and multi-level features fusion.
Smart Images

Figure CN118918511A_ABST
Abstract
Description
Technical Field
[0001] The invention belongs to the technical field of detection and recognition, and relates to a group behavior recognition method based on dual-path fusion of contrastive learning. Background Art
[0002] Group Activity Recognition (GAR) generally refers to the technology of automatically detecting, analyzing and understanding the collective activities, interactions or collaborative behavior patterns of groups (such as crowds, animal groups, vehicle groups, network user groups, etc.) in the fields of computer vision, machine learning and artificial intelligence. This technology has a wide range of applications in many fields, including: sociology, public safety, intelligent traffic management, video surveillance, sports analysis, animal behavior ecology research and crowd management. The key to group behavior recognition is that the technical means involve deep learning, computer vision, multi-object detection and tracking, behavior modeling state recognition, graph models, spatiotemporal sequence analysis, etc. With the advancement of AI technology, group behavior recognition is becoming more and more accurate, and it is of great value to urban management, event prediction, security strategies, traffic optimization, etc.
[0003] Group behavior recognition aims to identify the collective activities of multiple individuals in a video scene. Group behavior recognition has attracted extensive attention in the field of computer vision due to its wide applications in understanding social behavior, analyzing sports videos, and monitoring systems. Unlike traditional action recognition, group behavior recognition involves multiple individuals and complex interactions between individuals, so modeling the rich spatiotemporal contextual information between individuals is crucial to identifying group behavior.
[0004] Some existing group behavior recognition studies attempt to capture the temporal and spatial interactions between individuals using convolutional neural networks, recurrent neural networks, graph convolutional networks, and Transformers. For example, ARG uses the appearance and position information of individuals in each frame to construct a relationship graph. Some group behavior recognition methods model the temporal and spatial interactions of individuals separately. In addition, some group behavior recognition work has demonstrated the importance of integrating multi-scale features (such as subgroup level and scene level) to improve model performance.
[0005] However, the above methods ignore the importance of jointly considering spatiotemporal attention. In addition, the temporal and spatial interaction order between individuals in different group behaviors is different, so it is very important to adopt a complementary spatiotemporal modeling order to adapt to the diversity of group behaviors. Dual-AI jointly models the temporal and spatial interaction relationship of individuals through complementary spatiotemporal dual paths, and uses MAC loss to associate the individual interactions in the dual paths. However, in terms of dual path fusion, it simply adds the classification scores of the dual paths without in-depth exploration of different fusion methods of the dual branches. At the same time, the design of MAC loss is also relatively complex. In addition, it only focuses on the interaction at the individual instance level and ignores the interaction at the individual category level.
[0006] Therefore, a group behavior recognition method that fully explores the complex spatiotemporal interaction relationships between multiple individuals in the video is needed to solve the above problems. Summary of the invention
[0007] In order to solve the above technical problems, the present invention is specifically implemented through the following technical solutions.
[0008] A group behavior recognition method based on dual-path fusion of contrastive learning, comprising the following steps:
[0009] Step 1: Use Inception-v3 as the backbone network to extract feature maps of video clips;
[0010] Step 2: Based on N bounding boxes, ROI-Align is applied to each frame of the feature map extracted in step 1 to extract individual features;
[0011] Step 3: Use the fully connected layer to convert the dimension of the individual features extracted in step 2 into a D-dimensional vector to obtain the individual features
[0012] Step 4: Input the individual features obtained in step 3 into the dual path consisting of a temporal encoder and a spatial encoder to model the temporal-spatial and spatial-temporal interactive relationships of the individual features;
[0013] Step 5: Fuse the features of the dual paths consisting of the temporal encoder and the spatial encoder in step 4 to give full play to the complementary advantages of the dual paths;
[0014] Step 6: Distinguish individual features according to the real action labels and perform average pooling to obtain individual category-level features;
[0015] Step 7: Use an encoder with a multi-head attention mechanism to enhance the interaction between category features;
[0016] Step 8: By comparing the learning loss functions, the loss function of the positive sample pair is used to make the corresponding individual features on the two paths consistent and reduce noise interference, and the loss function of the negative sample pair is used to increase the differences between different individual categories and reduce inter-class confusion;
[0017] Step 9: Perform maximum pooling on the enhanced individual instance-level features and individual category-level features respectively to obtain group representation for group behavior classification.
[0018] Preferably, in step 5, a dual-path fusion network is formed based on the Transformer's time encoder and space encoder. By utilizing the ability of the self-attention mechanism in capturing long-range dependencies, the encoder can comprehensively capture the interaction relationship between individuals in the time and space domains, thereby enhancing the individual's spatiotemporal characteristics, and stacking the time encoder and space encoder in different orders to build a dual-path structure of space-time and time-space.
[0019] Preferably, the time encoder is expressed as The temporal encoder is to transform all individual features {X" n |n=1,…,N} are assembled together;
[0020] in:
[0021]
[0022] X” n =FFN(X' n ) (3)
[0023] Represents the features of the nth individual in T frames, and uses a temporal encoder based on a self-attention mechanism to extract the temporal dynamics of individuals across frames, thereby modeling the temporal evolution across frames to obtain temporal context information; SPE refers to spatial position encoding, which is used to add spatial distribution information of participants in the scene; W q ,W k ,W v is a learnable parameter with a shape of D×D; FFN is a feedforward network that can further enhance the learning ability of the encoder.
[0024] Preferably, the spatial encoder is expressed as The spatial encoder is to transform all individual features {X" t |t=1,…,T} are assembled together;
[0025] in:
[0026]
[0027] X”t =FFN(X' t ) (3-1)
[0028] Represents the features of the Nth individual in the Tth frame. The spatial encoder uses the time dimension as the batch dimension to model the spatial relationship between individuals in the frame; input is the feature vector of N individuals in the tth frame A multi-head self-attention mechanism is adopted to infer the spatial interactions between individuals within a frame, thereby enhancing the spatial features of individuals.
[0029] Preferably, in step 5, the feature fusion step of the dual paths composed of the temporal encoder and the spatial encoder is:
[0030] Step 5-1, based on the fusion of individual instance-level features, combined with the complementary advantages in capturing spatiotemporal clues, better achieve spatiotemporal feature enhancement; spatiotemporal feature X Fus It is expressed as:
[0031] X Fus =X TS +X ST (6)
[0032] In formula (6), X TS represents the space-time path, X ST represents the space-time dual path;
[0033] Step 5-2, based on the fusion of individual instance-level features, design an individual category-level interaction module to fuse the category-level features;
[0034] Step 5-3: Input includes the spatiotemporal features after fusion at the individual instance level and real-action tags;
[0035] Step 5-4: The features of N individuals in each frame Divide into M categories according to the labels, and average pool all individuals in each category to obtain category-level individual features
[0036] Step 5-5, the features of all frames Packaged In steps 5-6, an encoder with a multi-head self-attention mechanism is used to model the interactions between categories and obtain updated category-level features.
[0037] Preferably, in step 8, the loss function of the positive sample pair is expressed as: L Pos ;
[0038]
[0039] In formula (8) and formula (9), sim i,j represents the cosine similarity between positive sample pairs, ||·|| represents the L2 norm of the vector,
[0040] Preferably, the loss function of the negative sample pair is expressed as: L Neg ;
[0041]
[0042] In formula (10), s i,j represents the cosine similarity between negative sample pairs; L Neg Increase the distance between different categories of individuals.
[0043] Preferably, in step 9, the network is trained in an end-to-end manner to predict group behavior categories; individual and group classification uses cross entropy loss, and the formula is as follows:
[0044]
[0045] In formula (11), represents the prediction score of group behavior classification after pooling of individual instance-level features, represents the prediction score of group behavior classification after pooling of individual category-level features, represents the prediction score of individual action classification, y G and Represent the true labels of group behavior and individual action respectively; λ is a hyperparameter to balance these two terms;
[0046] Then, all losses are combined to train the framework:
[0047] L=L cls +L1 (12)
[0048] In formula (12), L1 represents the contrast loss function.
[0049] The beneficial effects of the present invention are:
[0050] 1. The present invention integrates complementary spatiotemporal and space-time contextual information and fuses multi-level features to effectively model the complex interactive relationships between individuals in video scenes. Therefore, the present invention can explore the complex spatiotemporal interactive relationships between multiple individuals in the video.
[0051] 2. The contrastive learning loss function designed by the present invention maintains the representation consistency of the same individual in the dual paths and expands the differences between individuals of different categories, thereby reducing individual category confusion and improving the effect of group behavior recognition. BRIEF DESCRIPTION OF THE DRAWINGS
[0052] Figure 1 It is a structural schematic diagram of a group behavior recognition method based on dual-path fusion of contrastive learning according to the present invention; DETAILED DESCRIPTION
[0053] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the relevant technologies in the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.
[0054] In order to explore the complex spatiotemporal relationship between participants in group behavior, this embodiment proposes a new dual-path fusion network based on contrastive learning, which enhances the spatiotemporal relationship between participants by fusing the complementary spatiotemporal features of the two paths. In addition, a contrastive loss function is designed to maintain the representation consistency between the corresponding individuals on the dual paths, while increasing the distinction between different individual categories. This section will introduce the framework of this embodiment in detail.
[0055] This embodiment is based on a dual-path fusion network of contrastive learning, which enhances the discriminative ability of features by integrating complementary features of different spatiotemporal paths. SE and TE represent the spatial encoder and the temporal encoder. In the individual category level interaction module, shapes of different colors represent individuals of different categories. In addition, Pos-loss is a positive pair loss, which is used to maintain the consistency of representation on the dual paths, while Neg-loss is a negative pair loss, which is used to expand the differences in representations of different categories.
[0056] like Figure 1 As shown, given a video clip with individual bounding box annotations, this embodiment first uses Inception-v3 as the backbone network to extract feature maps. Subsequently, based on N bounding boxes, this embodiment applies ROI-Align to the feature map of each frame to extract individual features. Next, a fully connected layer is used to convert the dimensions of the individual features into a D-dimensional vector.
[0057] Get individual characteristics After that, this embodiment inputs it into a dual path consisting of a time and space encoder to model the time-space and space-time interaction relationship of individual features. Different from the operations processed separately by the adopted dual paths, this embodiment fuses the features of the two paths to give full play to the complementary advantages. In addition, in order to fuse contextual information at different levels, this embodiment distinguishes individual features according to real action labels, and then performs average pooling to obtain individual category-level features. Subsequently, an encoder with a multi-head attention mechanism is used to enhance the interaction between category features. In addition, this embodiment designs a unique contrastive learning loss function, in which the loss of positive sample pairs is used to keep the corresponding individual features on the two paths consistent and reduce noise interference, and the loss of negative sample pairs is used to increase the differences between different individual categories and reduce inter-class confusion. Finally, the enhanced individual instance-level features and individual category-level features are respectively subjected to maximum pooling to obtain group representation for group behavior classification.
[0058] Dual-path space-time interaction
[0059] This embodiment uses the Transformer-based temporal and spatial encoders to form the dual-path fusion network of this embodiment. By utilizing the powerful ability of the self-attention mechanism in capturing long-range dependencies, the encoder can fully capture the interactive relationship of individuals in the temporal and spatial domains, thereby enhancing the temporal and spatial characteristics of individuals. Subsequently, this embodiment stacks the temporal and spatial encoders in different orders to build a dual-path structure of space-time and time-space.
[0060] Time encoder. Represents the features of the nth individual in T frames, and uses the temporal encoder (TE) based on the self-attention mechanism to extract the temporal dynamics of individuals across frames, thereby modeling the temporal evolution across frames to obtain temporal context information. The process can be expressed as:
[0061]
[0062] X″ n =FFN(X′ n ), (3)
[0063] SPE refers to spatial position encoding [5], which is used to add spatial distribution information of participants in the scene. W q ,W k ,W v is a learnable parameter with a shape of D×D. FFN is a standard feedforward network that can further enhance the learning ability of the encoder. Then, all individual features {X” n |n=1,…,N} assembled together
[0064] Spatial Encoder. The structure of the spatial encoder (SE) is similar to that of the temporal encoder, except that SE uses the temporal dimension as the batch dimension to model the spatial relationship between individuals within a frame. Input the feature vector of N individuals in the tth frame Similar to formulas (1) and (2), this embodiment uses a multi-head self-attention mechanism to infer the spatial interactions between individuals in the frame, thereby enhancing the spatial characteristics of the individuals. Then, the features {X” t |t=1,…,T} is combined into
[0065] Dual-path structure. Since the temporal and spatial encoders can capture temporal and spatial context information respectively, this embodiment stacks them to form a spatiotemporal path. Paths of different spatiotemporal orders play different roles in various group behaviors. Therefore, this embodiment constructs a temporal-spatial and spatiotemporal dual-path structure:
[0066] X TS =SE(X+TE(X)) (4)
[0067] X ST =TE(X+SE(X)).(5)
[0068] Individual category level interactions
[0069] Different from using dual-path classification and then summing the classification scores, this embodiment deeply explores the dual-branch feature fusion method. First, fusion is performed based on individual instance-level features. This enables this embodiment to combine their complementary advantages in capturing spatiotemporal clues and better achieve spatiotemporal feature enhancement:
[0070] X Fus =X TS +X ST . (6)
[0071] On the basis of fusing individual instance-level features, this embodiment further designs an individual category-level interaction module to fuse category-level features. Figure 1 The module in demonstrates the whole process. The input includes the spatiotemporal features after fusion at the individual instance level and real action labels. In this embodiment, the features of N individuals in each frame are Divide into M categories according to the labels, and average pool all individuals in each category to obtain category-level individual features Then, the features of all frames are Packaged Finally, this embodiment uses an encoder with a multi-head self-attention mechanism to model the interaction between categories and obtain updated category-level features. The specific process is the same as formula (2) and formula (3).
[0072] Network Optimization
[0073] For individual instance-level and individual category-level features, this embodiment also designs a new contrast loss function to guide the learning of dual-path features. This embodiment regards the dual-path features of the same individual as positive pairs and the pooled different individual category-level features as negative pairs:
[0074] L1=λ p L Pos +λ n L Neg , (7)
[0075] Here λ(·)w represents the weights of different terms.
[0076] In order to maintain the consistency of the representation of the same individual on the two paths and reduce noise, this embodiment designs a loss function L for the positive pair Pos :
[0077]
[0078] Among them, sim i,j represents the cosine similarity between positive sample pairs, ||·|| represents the L2 norm of the vector,
[0079] In order to avoid confusion between different individual categories, this embodiment designs a loss function L for negative pairs Neg :
[0080]
[0081] where s i,j represents the cosine similarity between negative sample pairs, and the calculation process is similar to formula (9). Neg Increase the distance between different categories of individuals.
[0082] The network of this embodiment is trained in an end-to-end manner to predict group behavior categories. The standard cross entropy loss is used for individual and group classification, and the formula is as follows:
[0083]
[0084] in represents the prediction score of group behavior classification after pooling of individual instance-level features, represents the prediction score of group behavior classification after pooling of individual category-level features, represents the prediction score of individual action classification, y G and Represent the true labels of group behavior and individual action respectively. λ is a hyperparameter to balance these two terms. Finally, this embodiment combines all losses to train the framework of this embodiment:
[0085] L=L cls +L1. (12)
[0086] In summary, the present invention integrates complementary spatiotemporal and space-time contextual information and fuses multi-level features to effectively model the complex interaction relationships between individuals in video scenes. Therefore, the present invention can explore the complex spatiotemporal interaction relationships between multiple individuals in the video, and the method of the present invention has broad application prospects.
[0087] It should be emphasized that the above are only preferred embodiments of the present invention and do not limit the present invention in any form. Any simple modifications, equivalent changes and modifications made to the above embodiments based on the technical essence of the present invention are still within the scope of the technical solution of the present invention.
Claims
1. A dual-path fusion group behavior recognition method based on contrastive learning, characterized in that: The following steps are involved: Step 1: Use Inception-v3 as the backbone network to extract feature maps of video clips; Step 2: Based on N bounding boxes, ROI-Align is applied to each frame of the feature map extracted in step 1 to extract individual features; Step 3: Use the fully connected layer to convert the dimension of the individual features extracted in step 2 into a D-dimensional vector to obtain the individual features Step 4: Input the individual features obtained in step 3 into the dual path consisting of a temporal encoder and a spatial encoder to model the temporal-spatial and spatial-temporal interactive relationships of the individual features; Step 5: Fuse the features of the dual paths consisting of the temporal encoder and the spatial encoder in step 4 to give full play to the complementary advantages of the dual paths; Step 6: Distinguish individual features according to the real action labels and perform average pooling to obtain individual category-level features; Step 7: Use an encoder with a multi-head attention mechanism to enhance the interaction between category features; Step 8: By comparing the learning loss functions, the loss function of the positive sample pair is used to make the corresponding individual features on the two paths consistent and reduce noise interference, and the loss function of the negative sample pair is used to increase the differences between different individual categories and reduce inter-class confusion; Step 9: Perform maximum pooling on the enhanced individual instance-level features and individual category-level features respectively to obtain group representation for group behavior classification.
2. The method for group behavior recognition based on dual-path fusion of contrastive learning according to claim 1 is characterized in that: In step 5, a dual-path fusion network is formed based on the temporal encoder and spatial encoder of Transformer. By utilizing the ability of the self-attention mechanism in capturing long-range dependencies, the encoder can comprehensively capture the interactive relationship between individuals in the temporal and spatial domains, thereby enhancing the temporal and spatial characteristics of individuals. The temporal encoder and the spatial encoder are stacked in different orders to build a dual-path structure of space-time and time-space.
3. The method for group behavior recognition based on dual-path fusion of contrastive learning according to claim 2 is characterized in that: The time encoder is represented as The temporal encoder is to transform all individual features {X″ n |n=1,…,N} are assembled together; in: X″ n =FFN(X′ n ) (3) Represent the features of the nth individual in T frames, and use a temporal encoder based on a self-attention mechanism to extract the temporal dynamics of individuals across frames, thereby modeling the temporal evolution across frames to obtain temporal context information; SPE refers to spatial position encoding, which is used to add spatial distribution information of participants in the scene; W q , W k ,W v is a learnable parameter with a shape of D×D; FFN is a feedforward network that can further enhance the learning ability of the encoder.
4. The method for group behavior recognition based on dual-path fusion of contrastive learning according to claim 3 is characterized in that: The spatial encoder is represented as The spatial encoder is to transform all individual features {X″ t |t=1,…,T} are assembled together; in: X″ t =FFN(X′ t ) (3-1) Represents the features of the Nth individual in the Tth frame. The spatial encoder uses the time dimension as the batch dimension to model the spatial relationship between individuals in the frame; input is the feature vector of N individuals in the tth frame A multi-head self-attention mechanism is adopted to infer the spatial interactions between individuals within a frame, thereby enhancing the spatial features of individuals.
5. The method for group behavior recognition based on dual-path fusion of contrastive learning according to claim 1, characterized in that: In step 5, the feature fusion step of the dual path composed of the temporal encoder and the spatial encoder is: Step 5-1, based on the fusion of individual instance-level features, combined with the complementary advantages in capturing spatiotemporal clues, better achieve spatiotemporal feature enhancement; spatiotemporal feature X Fus It is expressed as: X Fus =X TS +X ST (6) In formula (6), X TS represents the space-time path, X ST represents the space-time dual path; Step 5-2, based on the fusion of individual instance-level features, design an individual category-level interaction module to fuse the category-level features; Step 5-3: Input includes the spatiotemporal features after fusion at the individual instance level and real-action labels; Step 5-4: The features of N individuals in each frame Divide into M categories according to the labels, and average pool all individuals in each category to obtain category-level individual features Step 5-5, the features of all frames Packaged In steps 5-6, an encoder with a multi-head self-attention mechanism is used to model the interactions between categories and obtain updated category-level features.
6. The method for group behavior recognition based on dual-path fusion of contrastive learning according to claim 1, characterized in that: In step 8, the loss function of the positive sample pair is expressed as: L Pos ; In formula (8) and formula (9), sim i,j represents the cosine similarity between positive sample pairs, ||·|| represents the L2 norm of the vector, 7. The method for group behavior recognition based on dual-path fusion of contrastive learning according to claim 6, characterized in that: The loss function of the negative sample pair is expressed as: L Neg ; In formula (10), s i,j represents the cosine similarity between negative sample pairs; L Neg Increase the distance between different categories of individuals.
8. The method for preparing a group behavior recognition method based on dual-path fusion of contrastive learning according to claim 1, characterized in that: In step 9, the network is trained in an end-to-end manner to predict group behavior categories; individual and group classification uses cross entropy loss, and the formula is as follows: In formula (11), represents the prediction score of group behavior classification after pooling of individual instance-level features, represents the prediction score of group behavior classification after pooling of individual category-level features, represents the prediction score of individual action classification, y G and Represent the true labels of group behavior and individual actions respectively; λ is a hyperparameter that balances these two terms; Then, all losses are combined to train the framework: L=L cls +L1 (12) In formula (12), L1 represents the contrast loss function.
Citation Information
Cited By
Group behavior identification method based on 3D human body skeleton
CN120708158A
