Video group behavior recognition method considering dynamic salient frames and multi-granularity information
By employing dynamic salient frames and multi-granularity information, the problem of insufficient representation of individual behavioral features by video frame sampling methods is solved, achieving efficient video group behavior recognition, especially high-accuracy recognition on the Social-CAD dataset.
Patent Information
- Application Number
- CN202410348584.5
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-03-26
- Publication Date
- 2026-11-17
- Estimated Expiration
- 2044-03-26
AI Technical Summary
Existing video frame sampling methods have limited representational power for individual behavioral characteristics and poor recognition performance.
By employing dynamic salient frames and multi-granularity information, we identify individuals, subgroups, and groups in continuous video frames, calculate the contribution of individual behavior to changes over time, construct a subset of dynamic salient frames, extract individual temporal features, and perform group behavior recognition through message passing and a multilayer perceptron.
It improves the accuracy of video group behavior recognition, especially achieving a recognition accuracy of 96.7% on the Social-CAD dataset, and enhances the representation power of group behavior features.
Smart Images

Figure CN118196688B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of computer vision technology, and in particular to a video group behavior recognition method that takes into account dynamic salient frames and multi-granularity information. Background Technology
[0002] Video group behavior recognition is a crucial research area in computer vision, aiming to analyze the collective behavior of all individuals within a group. It can be applied to sports event analysis, abnormal behavior detection, and early warning systems. Compared to individuals, groups contain more information, such as changes in individual behavior over time and the mutual influence between individual behaviors; compared to a crowd, a group is a collection of individuals with related behaviors, exhibiting behavioral consistency or intentional relevance.
[0003] After locating groups in a video, temporal and spatial features can be used to characterize group behavior. One approach is to first characterize the changes in individual behavior over time as temporal features, then characterize the relationships between individuals as spatial features, and finally characterize group behavior by connecting these temporal and spatial features. Another approach is to characterize the relationships between individuals frame by frame as spatial features, then characterize the changes in group behavior over time as temporal features, and finally characterize group behavior by connecting these spatial and temporal features. Regardless of the approach used, the selection of video frames is a crucial factor. Currently, the main method is to segment the video chronologically, randomly extract frames within each interval, and perform temporal and spatial feature extraction, or spatial and temporal feature extraction, on all sampled frames in the video. The problem with this sampling method is that the extracted frames have limited representational power for individual behavioral features, resulting in poor recognition performance.
[0004] Therefore, existing technologies still need to be improved and enhanced. Summary of the Invention
[0005] In view of the shortcomings of the prior art, the purpose of this invention is to provide a video group behavior recognition method that considers dynamic salient frames and multi-granularity information, aiming to solve the problem that the sampling method in the prior art has a small representation power for the individual behavioral features of the extracted frames and poor recognition performance.
[0006] To achieve the above objectives, the present invention adopts the following technical solution:
[0007] Firstly, a video group behavior recognition method considering dynamic salient frames and multi-granularity information includes:
[0008] Step 1: Identify individuals, subgroups, and groups within consecutive video frames;
[0009] Step 2: Divide the video frame sequence at fixed intervals. Consider each individual in turn within each interval and calculate the contribution of each frame to the individual's behavior in terms of the change of individual behavior over time. Construct a dynamic salient frame subset. The dynamic salient frame subsets of all intervals constitute the dynamic salient frame set of the individual.
[0010] Step 3: Extract the temporal features of an individual from its dynamic salient frame set and use them as attributes of the individual;
[0011] Step 4: Calculate the correlation of all individual attributes within each subgroup and use it as a measure of the correlation between individual behaviors or intentions. Perform message passing to characterize the mutual influence between individual behaviors, and calculate the spatial features of the subgroups.
[0012] Step 5: Connect individual temporal features and subgroup spatial features as attributes of the subgroup, which are used to characterize the behavior of the subgroup and serve as subgroup features;
[0013] Step 6: Calculate the relevance of subgroup attributes within the group and use it as a measure of the relevance of behaviors or intentions between subgroups. Perform message passing to characterize the mutual influence between subgroup behaviors, and calculate the spatial characteristics of the group.
[0014] Step 7: Connect the temporal features from Step 3 and the group spatial features from Step 6, and use them as inputs to the multilayer perceptron to complete group behavior recognition.
[0015] Furthermore, determining individuals, subgroups, and groups within consecutive video frames includes:
[0016] Step 1.1: Consider each frame of the video, use a feature extraction network to calculate the feature map, use individual constraint boxes to define the feature map of an individual, normalize the size of the individual feature map and use the feature map to represent the individual, and obtain the feature map of all individuals in the frame.
[0017] Step 1.2: Use clustering to obtain the subgroup information and group information that the individual depends on.
[0018] Furthermore, step two includes:
[0019] Step 2.1: Represent the individual feature set of the video frame sequence using a triplet (N,T,F), where N is the number of individuals, T is the number of frames, and F is the feature map obtained in Step 1.1. i ,T i ,F i ) represents the set of individual features of the i-th video frame subsequence, where N i T represents the number of individuals. i For frames, F i For the feature maps obtained in step 1.1, the similarity between feature maps of the same person in different frames is calculated using formula (1):
[0020]
[0021] Where Q, K, and V correspond to the linearized F in the triplet. i d k For F i The length of the feature vector of a single person in a single frame after linearization. The purpose of the Softmax function is to ensure that the value is between 0 and 1.
[0022] Step 2.2: When extracting video frames at the individual granularity, select frames with low similarity, use formula (2) to achieve this effect, and calculate the dynamic salient frame subset:
[0023]
[0024] Among them, (QK) T ) -1 This characterizes the dissimilarity between feature maps of the same individual in different frames within the i-th video frame subsequence. For each individual in AttentionF, the M elements with the largest dissimilarity values are selected to form a dynamically significant subset of frames.
[0025] Step 2.3: The dynamic salient frame set of an individual is formed by the subset of dynamic salient frames of all intervals, represented by a triple (N, DT, DF), where DT is the number of dynamic salient frames and DF is the feature map of the corresponding dynamic salient frame in step 1.1.
[0026] Furthermore, step three includes:
[0027] Step 3.1: Linearize each feature map and add positional encoding to obtain gain features.
[0028] Step 3.2: Input the gain features into the feedforward network to extract nonlinear features, and use the residual structure to superimpose them, and then extract individual time features.
[0029] Furthermore, step four includes:
[0030] Step 4.1: Consider each subgroup, calculate the correlation of attributes of all individuals within each subgroup, and obtain the individual association matrix as a measure of the correlation between individual behaviors or intentions;
[0031] Step 4.2: Characterize the mutual influence between individual behaviors through message passing, complete the update of individual attributes, and then extract the spatial features of subgroups.
[0032] Furthermore, step six includes:
[0033] Considering groups, calculate the relevance of subgroup attributes within a group to obtain a subgroup association matrix, which serves as a measure of the correlation between behaviors or intentions among subgroups;
[0034] The message passing of formula (1) is used to characterize the mutual influence between subgroup behaviors, complete the subgroup attribute update, and then extract the group space features.
[0035] Furthermore, the method also includes:
[0036] During the training phase, loss functions are constructed at the individual, subgroup, and group levels, respectively, using cross-entropy as the loss function. The total loss, Loss, can be expressed as:
[0037] Loss = L i + L sg + L g (3)
[0038] Among them, L i L sg and L g These represent individual-level loss, subgroup-level loss, and group-level loss, respectively.
[0039] The technical solution adopted in this invention has the following beneficial effects:
[0040] In this application, video frame extraction uses individuals as the information granularity instead of groups. Dynamically significant frames are extracted to better represent changes in individual behavior over time and reduce information redundancy. When representing group behavior, subgroups are added between individuals and groups, leveraging the better behavioral consistency or intention of individuals within these subgroups. Figure 1 Based on the characteristics of consistency and clear semantic meaning of subgroups, this study uses information at three levels—individual, subgroup, and group—to characterize group behavior and enhance the representational power of group behavior features. On the Social-CAD group behavior dataset, using only RGB modal data, a recognition accuracy of 96.7% was achieved, improving recognition performance. Attached Figure Description
[0041] Figure 1 This is a schematic diagram illustrating the principle structure of a video group behavior recognition method that considers dynamic salient frames and multi-granularity information provided by the present invention. Detailed Implementation
[0042] To make the objectives, technical solutions, and effects of this invention clearer and more explicit, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the invention and are not intended to limit the invention.
[0043] It should be noted that when a component is referred to as being "fixed to" or "set on" another component, it can be directly on or indirectly on that other component. When a component is referred to as being "connected to" another component, it can be directly connected to or indirectly connected to that other component.
[0044] It should also be noted that the same or similar reference numerals in the accompanying drawings of the embodiments of the present invention correspond to the same or similar components; in the description of the present invention, it should be understood that if terms such as "upper," "lower," "left," "right," etc. indicate the orientation or positional relationship based on the orientation or positional relationship shown in the accompanying drawings, they are only for the convenience of describing the present invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, the terms used to describe positional relationships in the accompanying drawings are only for illustrative purposes and should not be construed as limiting the present patent. For those skilled in the art, the specific meaning of the above terms can be understood according to the specific circumstances.
[0045] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of indicated technical features. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of that feature. In the description of this invention, "a plurality of" means two or more, unless otherwise explicitly specified.
[0046] Please see Figure 1 In this embodiment, a video group behavior recognition method considering dynamic salient frames and multi-granularity information includes:
[0047] Step 1: Identify individuals, subgroups, and groups within consecutive video frames;
[0048] Specifically, considering each frame of the video, a feature extraction network is used to compute a feature map. Individual feature maps are defined using individual constraint boxes, the size of the individual feature maps is normalized, and these feature maps are used to represent the individuals, thus obtaining the feature maps of all individuals in that frame. Clustering is then used to obtain the subgroup information and group information that each individual depends on.
[0049] Step 2: Divide the video frame sequence at fixed intervals. Consider each individual in turn within each interval and calculate the contribution of each frame to the individual's behavior in terms of the change of individual behavior over time. Construct a dynamic salient frame subset. The dynamic salient frame subsets of all intervals constitute the dynamic salient frame set of the individual.
[0050] Specifically, the video frame sequence is divided at fixed intervals, and each sub-video frame sequence is processed sequentially in time. Within each sub-video frame sequence, each individual is considered, and the contribution of individual behavior in each frame as a function of time is calculated to construct a dynamically salient frame subset. The dynamically salient frame subsets of all intervals constitute the dynamically salient frame set of an individual.
[0051] Let the triplet (N,T,F) represent the individual feature set of the video frame sequence, where N is the number of individuals, T is the number of frames, and F is the feature map obtained in step one. i ,T i ,F i ) represents the set of individual features of the i-th video frame subsequence, where N i T represents the number of individuals. i For frames, F i For the feature maps obtained in step one, formula (1) can be used to calculate the similarity between feature maps of the same person in different frames.
[0052]
[0053] Where Q, K, and V correspond to the linearized F in the triplet. i d k For F i The length of the feature vector of a single person in a single frame after linearization is given by the function Softmax, which ensures that the value is between 0 and 1.
[0054] When extracting video frames at the individual granularity, it is natural to select frames with low similarity to reduce redundancy and preserve individual behavior changes over time to the maximum extent. Here, formula (2) is used to achieve this effect and to calculate the dynamic salient frame subset.
[0055]
[0056] Among them, (QK) T ) -1 This characterizes the dissimilarity between features of the same individual in different frames within the i-th video frame subsequence. For each individual in AttentionF, the M elements with the largest dissimilarity values are selected to form a dynamic salient frame subset. The dynamic salient frame set of an individual is composed of all interval dynamic salient frame subsets, represented by a triplet (N, DT, DF), where DT is the number of dynamic salient frames and DF is the feature map of the corresponding dynamic salient frame from step one.
[0057] Step 3: Extract the temporal features of an individual from its dynamic salient frame set and use them as attributes of the individual;
[0058] Specifically, for each individual, the corresponding feature map is obtained from the DF of the triplet. The feature map corresponding to the dynamic salient frame is considered in time sequence, linearized, and then formula (1) is executed and position encoding is added to maintain the order. The information obtained from the dynamic salient frame is connected in time sequence and the temporal features of the individual are extracted through the feedforward network. The temporal features are used as the attributes of the individual.
[0059] Step 4: Calculate the correlation of all individual attributes within each subgroup and use it as a measure of the correlation between individual behaviors or intentions. Perform message passing to characterize the mutual influence between individual behaviors, and calculate the spatial features of the subgroups.
[0060] Specifically, for each subgroup, the correlation of attributes of all individuals within each subgroup is calculated to obtain an individual association matrix, which serves as a measure of the correlation between individual behaviors or intentions. The mutual influence between individual behaviors is characterized through message passing, the individual attributes are updated, and then the spatial features of the subgroup are extracted.
[0061] Step 5: Connect individual temporal features and subgroup spatial features as attributes of the subgroup, which are used to characterize the behavior of the subgroup and serve as subgroup features;
[0062] Step 6: Calculate the relevance of subgroup attributes within the group and use it as a measure of the relevance of behaviors or intentions between subgroups. Perform message passing to characterize the mutual influence between subgroup behaviors, and calculate the spatial characteristics of the group.
[0063] Specifically, considering the group, the correlation of subgroup attributes is calculated within the group to obtain the subgroup association matrix, which serves as a measure of the correlation between behaviors or intentions among subgroups. Formula (1) is used for message passing to characterize the mutual influence between subgroup behaviors, complete the subgroup attribute update, and extract the group spatial features.
[0064] Step 7: Connect the temporal features from Step 3 and the group spatial features from Step 6, and use them as inputs to the multilayer perceptron to complete group behavior recognition.
[0065] Step 8: During the training phase, construct loss functions at the individual, subgroup, and group levels, respectively, using cross-entropy as the loss function. The total loss, Loss, can be expressed as...
[0066] Loss = L i + L sg + L g (3)
[0067] Among them, L i L sg and L g These represent individual-level loss, subgroup-level loss, and group-level loss, respectively.
[0068] In specific embodiments of the present invention, the following examples further demonstrate the specific steps implemented in this application:
[0069] (1) Considering each frame of the video, feature maps are calculated using ResNet50. Individual feature maps are defined using individual constraint boxes. The size of individual feature maps is normalized using the RoiAlign method, and these feature maps are used to represent individuals. This ensures the consistency of feature map sizes. All individual feature maps in the frame are obtained and stored in a tensor (B,P,T,F), where B corresponds to the video, P corresponds to the individual, T corresponds to the frame, and F corresponds to the feature map. Clustering is used to obtain the subgroup information and group information that the individual depends on, so that the group is represented by several subgroups. The group behavior representation is enhanced by considering subgroups between individuals and groups. The tensor (B,P,T,F) can be expanded to (B,G,GN,P,T,F), where B still corresponds to the video, G corresponds to the group, GN corresponds to the subgroups within the group, P corresponds to the individual within the subgroup, T corresponds to the frame, and F corresponds to the feature map.
[0070] (2) The video frame sequence is segmented at fixed intervals, and each sub-video frame sequence is processed sequentially in time. Within each sub-video frame sequence, each individual is considered, and multi-head attention is used to calculate the contribution of individual behavior in each frame in terms of the change of individual behavior over time. M frames with the highest dissimilarity are extracted to construct a dynamic salient frame subset. The dynamic salient frame subsets of all intervals constitute the dynamic salient frame set of an individual. Frames with low similarity are selected to reduce redundancy and preserve the change of individual behavior over time to the maximum extent.
[0071] (a) Consider (P,T) in (B,G,GN,P,T,F) i ,F), where T i Let represent the number of frames within the i-th interval. Keeping dimension P constant, let (T) i ,F) and (T i Multiplying the transposes of P, T, and F, the elements in the resulting tensor represent the similarity between any two frames of the same person. Inverting this tensor allows the elements to represent the dissimilarity between any two frames of the same person. Softmax is then applied to normalize the values in the resulting tensor, yielding the probability weight tensor (P, T). i ,T i This tensor describes the contribution of different frames to the same person's behavior within the i-th interval. The larger the dissimilarity value, the greater the difference, and the greater the contribution.
[0072] (b) Consider the probability weight tensor (P,T) i ,T i ), maintain dimensions (P,T) i The third dimension T remains unchanged. iThe second dimension, representing the dissimilarity between frames of the same person and other frames, is T. i The summation of the two dimensions indicates dissimilarity; the second dimension T is determined based on the summation result. i The dynamic saliency of each frame is calculated, with larger values indicating higher dynamic saliency. The frame association set of all individuals in the subgroup is stored in the tensor (P,T). i In this case, normalize the elements in the tensor.
[0073] (c) For each individual described in dimension P, select the index values of the top K frames with high dynamic saliency. These frames retain the temporal changes in individual behavior and weaken redundant information. The features of the dynamic saliency frames of all individuals are stored in tensor (P,K,F'), where F' is a part of F.
[0074] (d) The dynamic salient frame features of all intervals are stored in the tensor (B,G,GN,P,MK,F'). This preserves the complete temporal variation of all individuals.
[0075] (3) Linearize each feature map of F' in tensor (B,G,GN,P,MK,F') and add positional encoding to ensure the sequentiality between frames, to obtain tensor (B,G,GN,P,MK,F”) where F” contains positional encoding information. Extract individual temporal features from tensor (B,G,GN,P,MK,F”).
[0076] (a) Keeping the dimensions (B,G,GN,P) unchanged, multiply the transposes of (MK,F”) and (MK,F”) to obtain a tensor whose elements represent the similarity between any two frames of the same person. Softmax is then used to normalize the element values in the resulting tensor to obtain the probability weight tensor (P,MK,MK). Since this tensor is constructed based on the individual's dynamic salient frames, it describes the contribution of different frames to the individual's behavior; a higher similarity value indicates a greater contribution.
[0077] (b) Multiply (P,MK,F”) and (P,MK,MK) to complete the message passing and obtain the tensor (P,MK,F”). 3 Using residual structures, superimpose (P,MK,F”) and (P,MK,F”) 3 The resulting tensor is then input into a feedforward network to extract nonlinear features. Further extraction of these nonlinear features yields the tensor (P, MK, F). 4 Using residual structure superposition (P,MK,F) 3 ) and (P,MK,F 4 This allows us to obtain the temporal characteristics of individuals. Through these overlay operations, we can achieve better aggregation of information.
[0078] (c) The temporal characteristics of all individuals are stored in the tensor (B, G, GN, P, MK, F). t5 )middle.
[0079] (4) Consider the tensor (B,G,GN,P,MK,F”), obtain the tensor (B,G,GN,MK,P,F”) by exchanging dimensions P and MK, and extract the spatial features of the subgroup to consider the individual association.
[0080] (a) Keeping the dimensions (B,G,GN,MK) unchanged, multiply the transposes of (P,F”) and (P,F”) to obtain a tensor whose elements represent the associations between different individuals in the same subgroup. Softmax is used to normalize the element values in the resulting tensor to obtain the probability weight tensor (MK,P,P). This tensor describes the degree of association between different individuals in the subgroup; the higher the association value, the greater the contribution to the subgroup's behavior.
[0081] (b) Multiply (MK,P,F”) and (MK,P,GN) to complete the message passing and obtain the tensor (MK,P,F”). 3 Using residual structures, superimpose (MK,P,F”) and (MK,P,F”). 3 The resulting tensor is then input into a feedforward network to extract nonlinear features, obtaining the tensor (MK, P, F). 4 Using residual structure superposition (MK,P,F) 3 ) and (MK,P,F 4 This allows us to obtain the spatial features of subgroups. Through these overlay operations, we can achieve better aggregation of information.
[0082] (c) Spatial features of all subgroups are preserved in tensor (B, G, GN, MK, P, F) s5 )middle.
[0083] (5) Consider the time feature tensors (B, G, GN, P, MK, F) obtained in step (3). t5 The spatial feature tensors (B, G, GN, MK, P, F) obtained in step (4) and step (4) s5 For tensor (B,G,GN,MK,P,F) s5 Swap dimensions MK and P, and aggregate the tensor (B,G,GN,P,MK,F) by addition. t5 ) and (B,G,GN,P,MK,F s5 ), obtain the tensor (B,G,GN,P,MK,F) st1 This is used to characterize the behavior of subgroups. This completes the aggregation of temporal and spatial features within subgroups.
[0084] (6) Extract group spatial features to consider subgroup associations.
[0085] (a) For tensors (B,G,GN,P,MK,F) st1 Keeping the dimensions (B,G,GN,P) unchanged, let (MK,F) st1 ) and (MK,F st1 Multiplying the transposes of the given tensors yields tensors whose elements represent the similarity between any two frames of the same person. Softmax is then used to normalize the element values in the resulting tensor, yielding a probability weight tensor (P, MK, MK). (P, MK, F) st1 The message is passed by multiplying (P,MK,MK) and (P,MK,MK) to obtain the tensor (P,MK,F). t2 This is used to highlight the importance of key individuals. The temporal features of all individuals are stored in tensors (B, G, GN, P, MK, F). st2 )middle.
[0086] (b) Consider the tensor (B,G,GN,P,MK,F) st2 Keeping the dimensions (B, G, GN) unchanged, the features of the subgroups are aggregated by average pooling to obtain the tensor (B, G, GN, MK, F). st3 This allows for a constraint on the number of feature maps in the subgroups.
[0087] (c) Keep the dimensions (B,G,GN) unchanged, and change (MK,F) st3 ) and (MK,F st3 Multiplying the transposes of the given tensors yields tensors whose elements represent the relationships between different subgroups within the same group. Softmax is then used to normalize the element values in the resulting tensor, resulting in a probability weight tensor (GN, MK, MK). This tensor describes the degree of association between different subgroups within the same group; a higher association value indicates a greater contribution of the subgroup's behavior to the group's behavior.
[0088] (d) Put (GN,MK,F) st3 ) and (GN,MK,F st3 Multiplying them together completes message passing, obtaining the tensor (GN,MK,F) st4 Using residual structure superposition (GN,MK,F) st3 ) and (GN,MK,F st4 The resulting tensor is then input into a feedforward network to extract nonlinear features, yielding a tensor (GN,MK,F). st5 Using residual structure superposition (GN,MK,F) st4 ) and (GN,MK,F st5 This allows us to obtain the spatial characteristics of the group. Through these overlay operations, information aggregation is achieved.
[0089] (e) The spatial characteristics of the group are preserved in the tensor (B, G, GN, MK, F)st5 (In the text). Considering the variation in the number of subgroups, if insufficient, fill with -1.
[0090] (7) Consider the time feature tensors (B, G, GN, P, MK, F) obtained in step (3). t5 The spatial feature tensors (B, G, GN, MK, F) obtained in step (6) and step (6) st5 The algorithm combines the two by addition and uses a multilayer perceptron to perform behavior recognition. It uses ReLU as the activation function, uses Dropout to prevent overfitting and sets the dropout rate to 0.5, and uses the Softmax function to probabilize the output.
[0091] As can be seen from the above, the method used in this embodiment uses individuals as the information granularity instead of groups when extracting video frames. It extracts dynamic salient frames to better represent changes in individual behavior over time and reduces information redundancy. When representing group behavior, subgroups are added between individuals and groups, utilizing the better behavioral consistency or intention of individuals within the subgroups. Figure 1 Based on the characteristics of consistency and clear semantic meaning of subgroups, this study uses information at three levels—individual, subgroup, and group—to characterize group behavior and enhance the representational power of group behavior features. On the Social-CAD group behavior dataset, a recognition accuracy of 96.7% was achieved using only RGB modal data.
[0092] Other embodiments of the invention will readily occur to those skilled in the art upon consideration of the specification and practice of the solutions disclosed herein. This invention is intended to cover any variations, uses, or adaptations of the invention that follow the general principles of the invention and include common knowledge or customary techniques in the art not disclosed herein. The specification and examples are to be considered exemplary only, and the true scope and spirit of the invention are indicated by the claims.
Claims
1. A video group behavior recognition method considering dynamic salient frames and multi-granularity information, characterized in that, include: Step 1: Identify individuals, subgroups, and groups within consecutive video frames; Step 2: Divide the video frame sequence at fixed intervals. Consider each individual in turn within each interval and calculate the contribution of each frame to the individual's behavior in terms of the change of individual behavior over time. Construct a dynamic salient frame subset. The dynamic salient frame subsets of all intervals constitute the dynamic salient frame set of the individual. Step 3: Extract the temporal features of an individual from its dynamic salient frame set and use them as attributes of the individual; Step 4: Calculate the correlation of all individual attributes within each subgroup and use it as a measure of the correlation between individual behaviors or intentions. Perform message passing to characterize the mutual influence between individual behaviors, and calculate the spatial features of the subgroups. Step 5: Connect individual temporal features and subgroup spatial features as attributes of the subgroup, which are used to characterize the behavior of the subgroup and serve as subgroup features; Step 6: Calculate the relevance of subgroup attributes within the group and use it as a measure of the relevance of behaviors or intentions between subgroups. Perform message passing to characterize the mutual influence between subgroup behaviors, and calculate the spatial characteristics of the group. Step 7: Connect the temporal features from Step 3 and the group spatial features from Step 6, and use them as inputs to the multilayer perceptron to complete group behavior recognition.
2. The video group behavior recognition method considering dynamic salient frames and multi-granularity information according to claim 1, characterized in that, The determination of individuals, subgroups, and groups in consecutive video frames includes: Step 1.1: Consider each frame of the video, use a feature extraction network to calculate the feature map, use individual constraint boxes to define the feature map of an individual, normalize the size of the individual feature map and use the feature map to represent the individual, and obtain the feature map of all individuals in the frame. Step 1.2: Use clustering to obtain the subgroup information and group information that the individual depends on.
3. The video group behavior recognition method considering dynamic salient frames and multi-granularity information according to claim 1, characterized in that, Step two includes: Step 2.1: Represent the individual feature set of the video frame sequence using a triplet (N,T,F), where N is the number of individuals, T is the number of frames, and F is the feature map obtained in Step 1.
1. i ,T i ,F i ) represents the set of individual features of the i-th video frame subsequence, where N i T represents the number of individuals. i For frames, F i For the feature maps obtained in step 1.1, the similarity between feature maps of the same person in different frames is calculated using formula (1): Where Q, K, and V correspond to the linearized F in the triplet. i d k For F i The length of the feature vector of a single person in a single frame after linearization. The purpose of the Softmax function is to ensure that the value is between 0 and 1. Step 2.2: When extracting video frames at the individual granularity, select frames with low similarity, use formula (2) to achieve this effect, and calculate the dynamic salient frame subset: Among them, (QK) T ) -1 It represents the dissimilarity between feature maps of the same individual in different frames in the i-th video frame subsequence. Each individual in AttentionF is considered in turn, and the M elements with the largest dissimilarity values are selected to form a dynamic salient frame subset. Step 2.3: The dynamic salient frame set of an individual is formed by the subset of dynamic salient frames of all intervals, represented by a triple (N, DT, DF), where DT is the number of dynamic salient frames and DF is the feature map of the corresponding dynamic salient frame in step 1.
1.
4. The video group behavior recognition method considering dynamic salient frames and multi-granularity information according to claim 1, characterized in that, Step three includes: Step 3.1: Linearize each feature map and add positional encoding to obtain the gain features; Step 3.2: Input the gain features into the feedforward network to extract nonlinear features, and use the residual structure to superimpose them, and then extract individual time features.
5. The video group behavior recognition method considering dynamic salient frames and multi-granularity information according to claim 1, characterized in that, Step four includes: Step 4.1: Consider each subgroup, calculate the correlation of attributes of all individuals within each subgroup, and obtain the individual association matrix as a measure of the correlation between individual behaviors or intentions; Step 4.2: Characterize the mutual influence between individual behaviors through message passing, complete the update of individual attributes, and then extract the spatial features of subgroups.
6. The video group behavior recognition method considering dynamic salient frames and multi-granularity information according to claim 1 or 3, characterized in that, Step six includes: Considering groups, calculate the relevance of subgroup attributes within a group to obtain a subgroup association matrix, which serves as a measure of the correlation between behaviors or intentions among subgroups; The message passing of formula (1) is used to characterize the mutual influence between subgroup behaviors, complete the subgroup attribute update, and then extract the group space features.
7. The video group behavior recognition method considering dynamic salient frames and multi-granularity information according to claim 1, characterized in that, The method further includes: During the training phase, loss functions are constructed at the individual, subgroup, and group levels, respectively, using cross-entropy as the loss function. The total loss, Loss, is expressed as... Loss = L i + L sg + L g (3) Among them, L i L sg and L g These represent individual-level loss, subgroup-level loss, and group-level loss, respectively.
Citation Information
Patent Citations
Crowd behavior recognition method based on subgroup division and dynamic and static feature fusion
CN111274866A
Behavior recognition method and apparatus, electronic device, and storage medium
WO2023024438A1
