A dual-branch social group behavior recognition method and system based on appearance and refined trajectory dynamic features

Abstract

The application relates to the technical field of computer vision and machine learning, and discloses a double-branch social group behavior recognition method and system based on appearance and refined track dynamic features, wherein the method comprises the following steps: based on appearance features extracted by a backbone network, using a space and time encoder to refine and calculating cosine similarity to generate an appearance similarity matrix; refining a bounding box track, calculating track dynamic features, using an LSTM network to extract track features, calculating a time average distance, generating a track similarity matrix and predicting a group number; adding the appearance and track similarity matrices, using a spectral clustering algorithm based on a comprehensive similarity matrix and the predicted group number to perform group detection; and obtaining a group behavior category according to a group detection result. The application combines appearance and track features, improves the accuracy of social group behavior recognition through a double-branch network, and is suitable for a group behavior recognition task in a complex scene.

Description

Technical Field

[0001] This invention relates to the fields of computer vision and machine learning, specifically to the field of social group behavior recognition, and particularly to a dual-branch social group behavior recognition method and system based on appearance and refined trajectory dynamic features. Background Technology

[0002] Social group behavior recognition requires the accurate capture and analysis of interactions between individuals and patterns of group behavior. Traditional methods for social group behavior recognition mostly rely on the analysis of single features in images or videos, which has the following limitations:

[0003] 1. Limitations of Static Appearance Features: Existing behavior recognition methods typically rely on appearance features (such as image features, color, shape, etc.) to infer individual behavior. However, appearance features only reflect an individual's visual characteristics and fail to fully consider dynamic behavioral changes and interactions between people. Therefore, relying solely on appearance features for behavior recognition often suffers from insufficient accuracy in multi-target and complex scenarios, especially when individuals have similar appearances or are occluded, which can easily lead to misjudgments.

[0004] 2. The limited scope of trajectory features: Many existing methods focus on trajectory features, typically analyzing behavior by tracking changes in an individual's spatial location. However, the limited scope of trajectory features makes behavior identification difficult in crowded or complex scenarios. In particular, considering only positional changes while ignoring dynamic features such as velocity, acceleration, and direction of motion can lead to inaccurate identification of individual movement patterns, thus affecting the accurate identification of group behavior.

[0005] 3. Insufficient Integration of Appearance and Trajectory Information: Current social behavior recognition technologies mostly process appearance and trajectory information separately, lacking comprehensive utilization of these two types of information. Appearance information focuses on an individual's visual characteristics, while trajectory information emphasizes changes in an individual's spatial location. These two are complementary; processing only one branch can lead to information loss or error accumulation, thus affecting the accurate identification of group behavior. Especially when there is close interaction or spatial overlap among group members, it is often difficult to accurately identify group behavior based on only one type of feature. Summary of the Invention

[0006] The purpose of this invention is to provide a dual-branch social group behavior recognition method and system based on appearance and refined trajectory dynamic features. By processing appearance features and trajectory dynamic features in parallel, it can realize the detection and behavior recognition of social groups in videos.

[0007] To achieve the above objectives, the following technical solution is adopted:

[0008] In a first aspect, the present invention provides a method for identifying dual-branch social group behavior based on appearance and refined trajectory dynamic features, the method comprising:

[0009] Video frames are input into the backbone network to extract the appearance features of the target. The appearance features are then refined using a spatial encoder and a temporal encoder. Cosine similarity is calculated and an appearance similarity matrix is ​​generated.

[0010] Based on the trajectories of all targets in the video frame, the dynamic features of the target trajectory are calculated, and the trajectory features of the target are extracted from the dynamic features of the target trajectory using an LSTM network. The time average distance index between trajectory pairs is calculated, and the nodes and edges of the graph Transformer are constructed to generate a trajectory similarity matrix and predict the population size. The trajectory pair consists of the trajectory features of two targets.

[0011] The appearance similarity matrix and the trajectory similarity matrix are added together to obtain a comprehensive similarity matrix. Based on the comprehensive similarity matrix and the predicted number of groups, a clustering algorithm is used to perform clustering to obtain the group detection results.

[0012] Based on the group detection results, for each detected group, the appearance features of the targets contained in the group and the classification labels representing group behavior are input into the classification model, and the group behavior category is output.

[0013] Further, video frames are input into the backbone network to extract the appearance features of the target, and the appearance features are refined using a spatial encoder and a temporal encoder. Cosine similarity is calculated and an appearance similarity matrix is ​​generated, including:

[0014] For each target in the video frame, bounding box detection is performed. The video frame is input into the backbone network of ResNet structure to extract the initial feature map. The appearance features of each target are cropped out according to the bounding box through the RoIAlign operation.

[0015] The appearance features are input into the spatial encoder to capture spatial context information and obtain spatial features;

[0016] The spatial features are transposed, and the spatial dimension is used as the batch processing dimension. The transposed appearance features are input into the temporal encoder to capture temporal context information and obtain temporal features. The temporal encoder and the spatial encoder use the same multi-head self-attention algorithm to perform average pooling on the spatial features and temporal features in the temporal dimension, and aggregate the information in the temporal dimension to obtain refined appearance features.

[0017] Calculate the cosine similarity between the refined appearance features of each target to generate an appearance similarity matrix.

[0018] Furthermore, the spatial encoder employs a multi-head self-attention algorithm, calculated using the following formula:

[0019] MHSA(X t = Concat(head1,…,head) h W o

[0020]

[0021] Where X t Represents the appearance features in the t-th frame of a video sequence, head1 and head2. h These represent the outputs of the 1st and hth attention heads, respectively, where h is the number of attention heads, and W... o This represents the output projection matrix, used to map the concatenated multi-head attention output back to the original feature dimensions. and Let X represent the query, key, and value projection matrices of the i-th attention head, respectively; d represents the feature dimensions of the key and value; softmax represents the normalization exponential function; R represents the real number space; and D represents the input feature X. t The original feature dimension, MHSA(X) t ) represents the multi-head self-attention mechanism for input features X t The calculation results.

[0022] Furthermore, the cosine similarity between the refined appearance features of each target is calculated using the following formula to generate an appearance similarity matrix:

[0023]

[0024] Among them, S app (i,j) represents the appearance similarity matrix between the i-th and j-th targets, F i and F j Let represent the appearance features of the i-th and j-th targets respectively, and ∥·∥ represent the modulus operation.

[0025] Furthermore, based on the trajectories of all targets in the video frame, the dynamic features of the target trajectories are calculated, and the trajectory features of the targets are extracted from the dynamic features of the target trajectories using an LSTM network. The time-averaged distance index between trajectory pairs is calculated, and the nodes and edges of a graph Transformer are constructed to generate a trajectory similarity matrix and predict the group size, including:

[0026] The bounding box trajectories of all targets in the video frame are refined, and the dynamic features of the trajectory are calculated. These dynamic features include bounding box size (w, h), center point coordinates (x, y), velocity v, acceleration a, direction of motion θ, and aspect ratio r.

[0027] The trajectory dynamic features and the original bounding box coordinates are concatenated and input into an LSTM network to extract trajectory features; wherein, the trajectory features can characterize the long-term temporal dependencies in the trajectory dynamic features;

[0028] Calculate the average time distance between the trajectory features of each target to measure the similarity between trajectories;

[0029] The trajectory features output by the LSTM network are used as nodes of the graph Transformer, and the average time distance is used as the edge of the graph Transformer. The features are further refined through the graph Transformer to obtain refined trajectory node features and edge values, where the edge values ​​form the trajectory similarity matrix.

[0030] The refined trajectory node features from the graph Transformer are input into the MLP classification head to predict the number of groups in the scene.

[0031] Furthermore, the dynamic characteristics of the trajectory are calculated using the following formula:

[0032] w = x 2 -x 1 ,

[0033] y = y 2 -y 1 ,

[0034]

[0035] Where (x) 1 ,y 1 ) and (x 2 ,y 2 ) represents the coordinates of the top left and bottom right corners of the bounding box, arctan(·) is the operation to calculate the arctangent value, Δ(x,y) is the displacement of the center point between consecutive frames, Δt is the time interval between adjacent frames, Δv is the change in velocity, and Δx and Δy represent the displacement of the center point in the x and y directions, respectively.

[0036] Furthermore, the time-averaged distance between the trajectory features of each target is calculated to measure the similarity between trajectories, including:

[0037] Calculate the distance between the trajectory sequences corresponding to the i-th and j-th targets over time t. The calculation formula is as follows:

[0038]

[0039] in This represents the trajectory sequence corresponding to the i-th and j-th targets. Used to calculate the normalized GIoU distance between the i-th target and the j-th target at time t. This indicates that both trajectories exist at time t. This indicates that at least one trajectory does not exist;

[0040] The average time distance D(P) between the two trajectories is obtained by averaging the distance at each time step between the two trajectories. i ,P j The calculation formula is as follows:

[0041]

[0042] in and Let i and j represent the time steps in which trajectories i and j exist, respectively. If the two trajectories do not overlap at any time step, i.e. Then the distance is 0.

[0043] Further, the appearance similarity matrix and the trajectory similarity matrix are added together to obtain a comprehensive similarity matrix. Based on the comprehensive similarity matrix and the predicted group size, a clustering algorithm is used to perform clustering to obtain the group detection results, including:

[0044] The appearance similarity matrix and the trajectory similarity matrix are added together to obtain the comprehensive similarity matrix. The calculation formula is as follows:

[0045] S=λS app +(1-λ)S traj

[0046] Where S represents the comprehensive similarity matrix, S app S represents the appearance similarity matrix. traj denoted as the trajectory similarity matrix, where λ is the fusion weight parameter;

[0047] Based on the comprehensive similarity matrix and the predicted group size, a spectral clustering algorithm is used to perform clustering to obtain the group detection results.

[0048] Furthermore, based on the group detection results, for each detected group, the appearance features of the targets contained in the group and the classification labels representing group behavior are input into the classification model, and the group behavior category is output, including:

[0049] For each detected group, the appearance features within the group and a classification label representing group behavior are input into a Transformer network. The Transformer network captures the interaction relationships between individuals within the subgroup and aggregates this information into the classification label.

[0050] Input the classification labels output by the Transformer network into the classification head to obtain the group behavior categories.

[0051] Secondly, the present invention provides a dual-branch social group behavior recognition system based on appearance and refined trajectory dynamic features, the system comprising:

[0052] The appearance similarity matrix generation module is configured to input video frames into a backbone network, extract the appearance features of the target, refine the appearance features using a spatial encoder and a temporal encoder, calculate cosine similarity, and generate an appearance similarity matrix.

[0053] The trajectory similarity matrix generation module is configured to calculate the dynamic features of the target's trajectory based on the trajectories of all targets in the video frame, extract the target's trajectory features from the dynamic features of the target's trajectory using an LSTM network, calculate the time average distance index between trajectory pairs, construct the nodes and edges of a graph Transformer, generate a trajectory similarity matrix, and predict the group size; wherein, the trajectory pair consists of the trajectory features of two targets.

[0054] The group detection module is configured to add the appearance similarity matrix and the trajectory similarity matrix to obtain a comprehensive similarity matrix, and to perform clustering using a clustering algorithm based on the comprehensive similarity matrix and the predicted group size to obtain the group detection result;

[0055] The behavior recognition module is configured to, based on the group detection results, input the appearance features of the targets contained in the group and the classification labels representing the group behavior into the classification model for each detected group, and output the group behavior category.

[0056] The beneficial effects of this invention are:

[0057] This invention combines appearance branching and trajectory branching, comprehensively considering both the target's appearance features and the dynamic features in its motion trajectory, effectively improving the accuracy and robustness of social group behavior recognition. By refining the dynamic features in the trajectory and combining them with a spatiotemporal transformer to process appearance features, this invention can comprehensively capture and understand group behavior characteristics, reducing errors that may arise from processing a single feature, while also improving performance in complex environments. Attached Figure Description

[0058] Figure 1 The flowchart illustrates a two-branch social group behavior recognition method based on appearance and refined trajectory dynamic features according to an embodiment of the present invention. Figure 1 .

[0059] Figure 2The flowchart illustrates a two-branch social group behavior recognition method based on appearance and refined trajectory dynamic features according to an embodiment of the present invention. Figure 2 .

[0060] Figure 3 A flowchart illustrating the appearance similarity matrix generation process according to an embodiment of the present invention is shown.

[0061] Figure 4 A flowchart illustrating the generation of a trajectory similarity matrix and prediction of population size according to an embodiment of the present invention is shown.

[0062] Figure 5 A flowchart of a group detection process according to an embodiment of the present invention is shown.

[0063] Figure 6 A structural diagram of a dual-branch social group behavior recognition system based on appearance and refined trajectory dynamic features according to an embodiment of the present invention is shown. Detailed Implementation

[0064] The following specific examples illustrate the implementation of the present invention. Those skilled in the art can easily understand other advantages and effects of the present invention from the content disclosed in this specification. The present invention can also be implemented or applied through other different specific embodiments, and various details in this specification can also be modified or changed based on different viewpoints and applications without departing from the spirit of the present invention. It should be noted that, unless otherwise specified, the following embodiments and features described therein can be combined with each other.

[0065] The specific embodiments of the present invention will be described in further detail below with reference to the accompanying drawings and examples.

[0066] This invention provides a method for identifying dual-branch social group behavior based on appearance and refined trajectory dynamic features, such as... Figure 1 The diagram shows the flow of this method. Figure 1 This method, based on input video frames, processes the frames through two branch tasks. One branch task obtains the appearance similarity matrix, and the other branch task obtains the trajectory similarity matrix and the predicted group size N. After processing by these two branch tasks, a comprehensive similarity matrix is ​​calculated based on the appearance and trajectory similarity matrices. Then, based on the comprehensive similarity matrix and the predicted group size N, a spectral clustering algorithm is used for group detection. This involves counting individual characteristics (appearance features) and classification tokens within the group and feeding them into a Transformer. The Transformer captures interaction relationships, and the classification head outputs the group behavior category.

[0067] Specifically, such as Figure 2The diagram shows the flowchart of a two-branch social group behavior recognition method based on appearance and refined trajectory dynamic features. Figure 2 The dual-branch social group behavior recognition method based on appearance and refined trajectory dynamic features is implemented by the following steps S100-S400.

[0068] S100. Input the video frames into the backbone network, extract the appearance features of the target, refine the appearance features using a spatial encoder and a temporal encoder, calculate the cosine similarity, and generate an appearance similarity matrix.

[0069] It should be noted that the term "video frame" in this article refers to a continuous image extracted frame by frame from video stream data containing a target. The target is generally present in at least one image, and usually multiple. The target refers to the elements constituting a social group. A social group is a group of organisms of the same kind, such as a social group of humans or a social group of animals. Animals refer to those with social habits and certain social behaviors, such as primates (chimpanzees, baboons, etc.), canines (wolves, African wild dogs, etc.), marine mammals (dolphins, killer whales, etc.), social insects (bees, ants, etc.), and birds (penguins, crows, etc.). Methods of acquiring video stream data include, but are not limited to, legally capturing images of the target using photographic equipment.

[0070] In some embodiments, such as Figure 3 As shown, step S100 includes the following steps:

[0071] S101. Detect bounding boxes for each target in the video frame. Input the video frame into the backbone network of the ResNet structure, extract the initial feature map, and use the RoIAlign operation to crop out the appearance features of each target based on the bounding box.

[0072] For example, bounding box detection for each target in a video frame can be performed using a two-stage detector (such as Faster R-CNN) or a single-stage detector. A two-stage detector generates candidate boxes using a Region Proposal Network (RPN). The RPN slides a window across the feature map, predicting the bounding box offset and foreground probability for each anchor point. A single-stage detector directly predicts the bounding box coordinates and class probability for each location on the feature map, without requiring an RPN. Two-stage detectors offer high accuracy and are suitable for complex scenes; single-stage detectors are fast and suitable for real-time detection. The specific method chosen depends on the actual requirements. Both two-stage and single-stage detectors output the bounding box coordinates and confidence score for each target.

[0073] The ResNet backbone network addresses the vanishing gradient problem during deep network training by introducing residual modules. Its formula is y = F(x, W). i)+x; where x is the input, y is the output, F is the residual function, and W is the residual function. i The weights are used for the input video frames. A typical ResNet architecture (such as ResNet50) consists of four stages (Stage 1 to Stage 4), progressively extracting semantic features from low to high levels. After the input video frame passes through ResNet, the output high-dimensional feature map serves as the initial feature map. The initial feature map extracted by the backbone network of the ResNet structure has powerful multi-scale feature representation capabilities, making it suitable for detecting targets of different sizes. Deeper networks retain rich contextual information, improving the robustness of small target detection.

[0074] The RoIAlign operation is a region of interest feature alignment operation. Its inputs are candidate bounding box coordinates (i.e., predicted bounding box coordinates) and an initial feature map. The specific operation is as follows:

[0075] Coordinate mapping: Map the bounding box from the original image coordinates to the feature map coordinates (the downsampling step size needs to be considered, such as ResNet's 32x downsampling).

[0076] Region division: Divide the candidate region into K×K uniformly distributed sampling points (e.g., 7×7).

[0077] Bilinear interpolation: For each sampling point, interpolation is performed based on the values ​​of the four surrounding feature map positions to avoid quantization errors.

[0078] Pooling output: Max pooling or average pooling is performed on the interpolation results in each sub-region to obtain features of a fixed size (e.g., 7×7×2048).

[0079] The RoIAlign operation can preserve subpixel-level accuracy and improve the alignment quality of small target features (such as improving the accuracy of pedestrian detection).

[0080] S102. Input the appearance features into the spatial encoder to capture spatial context information and obtain spatial features.

[0081] For example, the spatial encoder can be either a CNN or a VisionTransformer, which is used to model the visual content within a single frame, identify objects, scenes, and local details, effectively eliminate spatial redundancy information, and map the original pixels to a high-dimensional semantic space.

[0082] S103. Transpose the spatial features and use the spatial dimension as the batch processing dimension. Input the transposed appearance features into the temporal encoder to capture temporal context information and obtain temporal features. The temporal encoder and the spatial encoder use the same multi-head self-attention algorithm to perform average pooling on the spatial features and temporal features in the temporal dimension, aggregate the information in the temporal dimension, and obtain refined appearance features.

[0083] In this embodiment, the purpose of transposing the spatial features and using the spatial dimension as the batch processing dimension is to decouple the spatial location into independent "samples", so that the time encoder can process the temporal changes of all spatial locations in parallel, thereby avoiding the coupling between the spatial dimension and the time dimension and simplifying the computational complexity of time modeling.

[0084] For example, a temporal encoder can be one or a combination of RNN, LSTM, GRU, and Transformer. For instance, a temporal encoder can use the Transformer's Self-Attention mechanism to compute dependencies along the temporal dimension. The temporal features obtained through the temporal encoder are then integrated with the global temporal context.

[0085] This embodiment uses a combination of spatial and temporal encoders to model dynamic changes, such as target motion and target state evolution, thereby capturing long-term dependencies, such as the continuity of motion across multiple frames in a video.

[0086] In some embodiments, the spatial encoder employs a multi-head self-attention algorithm, calculated as follows:

[0087] MHSA(X t = Concat(head1,…,head) h W o

[0088]

[0089] Where X t Represents the appearance features in the t-th frame of a video sequence, head1 and head2. h These represent the outputs of the 1st and hth attention heads, respectively, where h is the number of attention heads, and W... o This represents the output projection matrix, used to map the concatenated multi-head attention output back to the original feature dimensions. and Let X represent the query, key, and value projection matrices of the i-th attention head, respectively; d represents the feature dimensions of the key and value; softmax represents the normalization exponential function; R represents the real number space; and D represents the input feature X. t The original feature dimension, MHSA(X) t ) represents the multi-head self-attention mechanism for input features X t The calculation results.

[0090] S104. Calculate the cosine similarity between the refined appearance features of each target and generate an appearance similarity matrix.

[0091] For example, assuming N targets are detected, an N×N appearance similarity matrix is ​​generated, where each element in the matrix represents the appearance similarity between two people.

[0092] In some embodiments, the cosine similarity between the refined appearance features of each target is calculated using the following formula to generate an appearance similarity matrix:

[0093]

[0094] Among them, S app (i,j) represents the appearance similarity matrix between the i-th and j-th targets, F i and F j Let represent the appearance features of the i-th and j-th targets respectively, and ∥·∥ represent the modulus operation.

[0095] S200. Based on the trajectories of all targets in the video frame, calculate the dynamic features of the target trajectory, and use an LSTM network to extract the target trajectory features from the dynamic features of the target trajectory. Calculate the time average distance index between trajectory pairs and construct the nodes and edges of the graph Transformer to generate a trajectory similarity matrix and predict the population size; wherein, a trajectory pair consists of the trajectory features of two targets.

[0096] In some embodiments, such as Figure 4 As shown, step S200 includes the following steps:

[0097] S201. Refine the bounding box trajectories of all targets in the video frame and calculate the trajectory dynamic features, which include bounding box size (w,h), center point coordinates (x,y), velocity v, acceleration a, motion direction θ, and aspect ratio r.

[0098] In some embodiments, the dynamic characteristics of the trajectory are calculated using the following formula:

[0099] w = x 2 -x 1 ,

[0100] y = y 2 -y 1 ,

[0101]

[0102] Where (x) 1 ,y 1 ) and (x 2 ,y 2) represents the coordinates of the top left and bottom right corners of the bounding box, arctan(·) is the operation to calculate the arctangent value, Δ(x,y) is the displacement of the center point between consecutive frames, Δt is the time interval between adjacent frames, Δv is the change in velocity, and Δx and Δy represent the displacement of the center point in the x and y directions, respectively.

[0103] S202. The trajectory dynamic features and the original bounding box coordinates are concatenated and input into the LSTM network to extract the trajectory features; the trajectory features can characterize the long-term time dependencies in the trajectory dynamic features.

[0104] For example, the original bounding box coordinates include the target position information at each time step, which can be represented as (x, y, w, h), where x and y are the coordinates of the target center point, and w and h are the width and height of the bounding box. The trajectory dynamic features are temporal change features derived from the bounding box coordinates. The specific calculation method has been described in step S201, so it will not be elaborated here. Before concatenating the trajectory dynamic features and the original bounding box coordinates, a data normalization operation is required. The purpose of the data normalization operation is to normalize the trajectory dynamic features and the original bounding box coordinates to the same dimension (such as [0,1] or [-1,1]), resulting in normalized trajectory dynamic features and original bounding box coordinates. For ease of description, these are defined as bounding box coordinates B, respectively. t and dynamic features F t Set the bounding box coordinates B t and dynamic features F t The concatenation yields a joint feature vector Input. t The joint feature vector Input t The input is fed into the LSTM network. The LSTM network consists of at least one LSTM unit, which processes the sequence information through a gating mechanism (input gate, forget gate, output gate) and finally outputs the trajectory features.

[0105] S203. Calculate the average time distance between the trajectory features of each target to measure the similarity between trajectories.

[0106] In some embodiments, for the trajectory sequences corresponding to the i-th and j-th targets, their distance at time t is first calculated. That is, the distance between the bounding boxes corresponding to individual i and individual j in frame t. The calculation formula is as follows:

[0107]

[0108] Where P i P j This represents the trajectory sequence corresponding to the i-th and j-th targets. This is used to calculate the normalized GIoU distance between individuals i and j at time t. If both trajectories exist at time t (i.e., ...), then... If at least one trajectory does not exist (i.e., GIoU distance is used), then the GIoU distance is used. If the distance is 1, then the distance is set to the maximum value of 1. Otherwise, the distance is set to the minimum value of 0.

[0109] The average time distance D(P) between the two trajectories is obtained by averaging the distance at each time step between the two trajectories. i ,P j The calculation formula is as follows:

[0110]

[0111] in and Let i and j represent the time steps in which trajectories i and j exist, respectively. If the two trajectories do not overlap at any time step, i.e. Then the distance is 0.

[0112] S204. The trajectory features output by the LSTM network are used as nodes of the graph Transformer, and the average time distance is used as the edge of the graph Transformer. The features are further refined through the graph Transformer to obtain refined trajectory node features and edge values, where the edge values ​​form the trajectory similarity matrix.

[0113] It should be noted that Graph Transformer is a deep learning framework that combines the self-attention mechanism of the Transformer model with graph-structured data, aiming to efficiently capture the complex relationships between nodes in a graph.

[0114] S205. Input the refined trajectory node features of the graph Transformer into the MLP classification head to predict the number of groups in the scene.

[0115] S300. Add the appearance similarity matrix and the trajectory similarity matrix to obtain the comprehensive similarity matrix. Based on the comprehensive similarity matrix and the predicted number of groups, use a clustering algorithm to perform clustering to obtain the group detection results.

[0116] In some embodiments, such as Figure 5 As shown, step S300 includes the following steps:

[0117] S301. Add the appearance similarity matrix and the trajectory similarity matrix to obtain the comprehensive similarity matrix. The calculation formula is as follows:

[0118] S=λS app +(1-λ)S traj

[0119] Where S represents the comprehensive similarity matrix, Sapp S represents the appearance similarity matrix. traj Let λ represent the trajectory similarity matrix, and λ be the fusion weight parameter.

[0120] S302. Based on the comprehensive similarity matrix and the predicted number of groups, the spectral clustering algorithm is used to perform clustering to obtain the group detection results.

[0121] For example, based on the comprehensive similarity matrix and the predicted group size, the specific implementation steps of using the spectral clustering algorithm to perform clustering and obtain the group detection results are as follows:

[0122] S3021. Calculate the normalized Laplacian matrix based on the comprehensive similarity matrix. The calculation formula is as follows:

[0123] D=diag∑S

[0124] L = ID -1 / 2 SD -1 / 2

[0125] Where D is the degree matrix, L is the normalized Laplace matrix, and I is the identity matrix.

[0126] S3022. Perform eigenvalue decomposition on the normalized Laplacian matrix. The decomposition formula is L. sym =UΛU T Select the eigenvectors U corresponding to the K smallest eigenvalues. K Where K is the predicted population size.

[0127] S3033, Decompose the eigenvector U obtained from the decomposition K Normalization is performed to obtain the normalized eigenvector U. norm .

[0128] S3034, Normalize the eigenvector U norm As input, the predicted population size is used as the number of clusters. The K-means algorithm is applied to perform clustering, and the population labels of the samples are output as {y1,y2,…,yN},yi∈{1,2,…,K}.

[0129] S400. Based on the group detection results, for each detected group, the appearance features of the targets contained in the group and the classification labels representing the group behavior are input into the classification model, and the group behavior category is output.

[0130] In some embodiments, step S400 includes the following steps:

[0131] S401. For each detected group, input the individual characteristics of the group members and a classification token representing group behavior into a Transformer network. The Transformer network captures the interaction relationships between individuals within the subgroup and aggregates this information into the classification token. Assuming group 1 contains 3 people, the input features are 3 individual characteristics and 1 classification token. The output is the classification token after aggregating the information.

[0132] S402. Input the classification token output by the Transformer network into the classification head composed of the MLP network to obtain the group behavior category. For example, the classification head may identify the behavior of one group as "queueing" and the behavior of another group as "gathering and talking".

[0133] This invention also provides a dual-branch social group behavior recognition system based on appearance and refined trajectory dynamic features. Please refer to [link to relevant documentation]. Figure 6 The system includes:

[0134] The appearance similarity matrix generation module 601 is configured to input video frames into a backbone network, extract the appearance features of the target, refine the appearance features using a spatial encoder and a temporal encoder, calculate cosine similarity, and generate an appearance similarity matrix.

[0135] The trajectory similarity matrix generation module 602 is configured to calculate the dynamic features of the trajectory of the target based on the trajectories of all targets in the video frame, extract the trajectory features of the target from the dynamic features of the target trajectory using an LSTM network, calculate the time average distance index between trajectory pairs and construct the nodes and edges of the graph Transformer, generate a trajectory similarity matrix and predict the group size; wherein, the trajectory pair consists of the trajectory features of two targets.

[0136] The group detection module 603 is configured to add the appearance similarity matrix and the trajectory similarity matrix to obtain a comprehensive similarity matrix, and to perform clustering using a clustering algorithm based on the comprehensive similarity matrix and the predicted group size to obtain the group detection result;

[0137] The behavior recognition module 604 is configured to, based on the group detection results, input the appearance features of the targets contained in the group and the classification labels representing the group behavior into the classification model for each detected group, and output the group behavior category.

[0138] It should be noted that the dual-branch social group behavior recognition system based on appearance and refined trajectory dynamic features belongs to the same technical concept as the previously described method, and has the same technical principles and beneficial effects, so it will not be repeated here.

[0139] The above embodiments are only used to illustrate the present invention and are not intended to limit the present invention. Those skilled in the art can make various changes and modifications without departing from the spirit and scope of the present invention. Therefore, all equivalent technical solutions also fall within the scope of the present invention, and the patent protection scope of the present invention should be defined by the claims.

Claims

1. A dual-branch social group behavior recognition method based on appearance and refined trajectory dynamic features, characterized in that, The method includes: Video frames are input into the backbone network to extract the appearance features of the target. The appearance features are then refined using a spatial encoder and a temporal encoder. Cosine similarity is calculated and an appearance similarity matrix is ​​generated. Based on the trajectories of all targets in the video frame, the dynamic features of the target trajectory are calculated, and the trajectory features of the target are extracted from the dynamic features of the target trajectory using an LSTM network. The time average distance index between trajectory pairs is calculated, and the nodes and edges of the graph Transformer are constructed to generate a trajectory similarity matrix and predict the population size. The trajectory pair consists of the trajectory features of two targets. The appearance similarity matrix and the trajectory similarity matrix are added together to obtain a comprehensive similarity matrix. Based on the comprehensive similarity matrix and the predicted number of groups, a clustering algorithm is used to perform clustering to obtain the group detection results. Based on the group detection results, for each detected group, the appearance features of the targets contained in the group and the classification labels representing group behavior are input into the classification model, and the group behavior category is output. Based on the trajectories of all targets in the video frame, the dynamic features of the target trajectories are calculated. An LSTM network is then used to extract the target trajectory features from these dynamic features. The time-averaged distance between trajectory pairs is calculated, and nodes and edges of a graph Transformer are constructed. A trajectory similarity matrix is ​​generated, and the population size is predicted, including: The bounding box trajectories of all targets in the video frame are refined, and the dynamic features of the trajectory are calculated. The dynamic features of the trajectory include the bounding box size ( , ), center point coordinates ,speed acceleration Direction of movement and aspect ratio : The trajectory dynamic features and the original bounding box coordinates are concatenated and input into an LSTM network to extract trajectory features; wherein, the trajectory features can characterize the long-term temporal dependencies in the trajectory dynamic features; Calculate the average time distance between the trajectory features of each target to measure the similarity between trajectories; The trajectory features output by the LSTM network are used as nodes of the graph Transformer, and the average time distance is used as the edge of the graph Transformer. The features are further refined through the graph Transformer to obtain refined trajectory node features and edge values, where the edge values ​​form the trajectory similarity matrix. The refined trajectory node features of the graph Transformer are input into the MLP classification head to predict the number of groups in the scene. Calculate the average time distance between the trajectory features of each target to measure the similarity between trajectories, including: Calculate the first The and the first The trajectory sequence corresponding to each target in time Distance on The calculation formula is as follows: in , Indicates the first The and the first The trajectory sequence corresponding to each target , used to calculate the first The first goal and the first One goal is Normalized GIoU distance at time, Indicates the time of two trajectories They all exist. 1 indicates that at least one trajectory does not exist; The average time distance between the two trajectories is obtained by averaging the distance at each time step between the two trajectories. The calculation formula is as follows: in and Representing trajectories and If two trajectories do not overlap at a given time step, that is... If the distance is 0, then the distance is 0.

2. The method as described in claim 1, characterized in that, Video frames are input into the backbone network to extract the appearance features of the target. These features are then refined using a spatial encoder and a temporal encoder. Cosine similarity is calculated, and an appearance similarity matrix is ​​generated, including: For each target in the video frame, bounding box detection is performed. The video frame is input into the backbone network of ResNet structure to extract the initial feature map. The appearance features of each target are cropped out according to the bounding box through the RoIAlign operation. The appearance features are input into the spatial encoder to capture spatial context information and obtain spatial features; The spatial features are transposed, and the spatial dimension is used as the batch processing dimension. The transposed appearance features are input into the temporal encoder to capture temporal context information and obtain temporal features. The temporal encoder and the spatial encoder use the same multi-head self-attention algorithm to perform average pooling on the spatial features and temporal features in the temporal dimension, and aggregate the information in the temporal dimension to obtain refined appearance features. Calculate the cosine similarity between the refined appearance features of each target to generate an appearance similarity matrix.

3. The method as described in claim 2, characterized in that, The spatial encoder uses a multi-head self-attention algorithm, and the calculation formula is as follows: in Indicates the first video sequence Appearance features in the frame, and They represent the 1st and the 2nd respectively. The output of each attention head, For the number of attention heads, This represents the output projection matrix, used to map the concatenated multi-head attention output back to the original feature dimensions. , and They represent the first The projection matrix of the query, key, and value of each attention head. The feature dimensions representing keys and values, Represents the normalized exponential function, Represents the space of real numbers. Representing input features The original feature dimension, This indicates that the multi-head self-attention mechanism is effective for input features. The calculation results.

4. The method as described in claim 1, characterized in that, The cosine similarity between the refined appearance features of each target is calculated using the following formula to generate an appearance similarity matrix: in, Indicates the first The and the first Appearance similarity matrix of each target and They represent the first The and the first The appearance characteristics of the target This indicates a modulo operation.

5. The method as described in claim 1, characterized in that, The dynamic characteristics of the trajectory are calculated using the following formula: in and These are the coordinates of the top-left and bottom-right corners of the bounding box, respectively. To obtain the arctangent value, The displacement of the center point between consecutive frames. The time interval between adjacent frames. The change in velocity, and These represent the displacements of the center point in the x and y directions, respectively.

6. The method as described in claim 1, characterized in that, The appearance similarity matrix and trajectory similarity matrix are added together to obtain a comprehensive similarity matrix. Based on the comprehensive similarity matrix and the predicted group size, a clustering algorithm is used to perform clustering to obtain the group detection results, including: The appearance similarity matrix and the trajectory similarity matrix are added together to obtain the comprehensive similarity matrix. The calculation formula is as follows: in Represents the comprehensive similarity matrix. Represents the appearance similarity matrix. Represents the trajectory similarity matrix. For fusion weight parameters; Based on the comprehensive similarity matrix and the predicted group size, a spectral clustering algorithm is used to perform clustering to obtain the group detection results.

7. The method as described in claim 1, characterized in that, Based on the group detection results, for each detected group, the appearance features of the targets contained in the group and the classification labels representing group behavior are input into the classification model, and the group behavior category is output, including: For each detected group, the appearance features within the group and a classification label representing the group's behavior are input into the Transformer network; the Transformer network captures the interaction relationships between individuals within the subgroup and aggregates this information into the classification label; Input the classification labels output by the Transformer network into the classification head to obtain the group behavior categories.

8. A dual-branch social group behavior recognition system based on appearance and refined trajectory dynamic features, characterized in that, The system for implementing the method as described in any one of claims 1 to 7 comprises: The appearance similarity matrix generation module is configured to input video frames into a backbone network, extract the appearance features of the target, refine the appearance features using a spatial encoder and a temporal encoder, calculate cosine similarity, and generate an appearance similarity matrix. The trajectory similarity matrix generation module is configured to calculate the dynamic features of the target's trajectory based on the trajectories of all targets in the video frame, extract the target's trajectory features from the dynamic features of the target's trajectory using an LSTM network, calculate the time average distance index between trajectory pairs, construct the nodes and edges of a graph Transformer, generate a trajectory similarity matrix, and predict the group size; wherein, the trajectory pair consists of the trajectory features of two targets. The group detection module is configured to add the appearance similarity matrix and the trajectory similarity matrix to obtain a comprehensive similarity matrix, and to perform clustering using a clustering algorithm based on the comprehensive similarity matrix and the predicted group size to obtain the group detection result; The behavior recognition module is configured to, based on the group detection results, input the appearance features of the targets contained in the group and the classification labels representing the group behavior into the classification model for each detected group, and output the group behavior category.

Images