A social grouping method for large scene video

Abstract

The application discloses a social grouping method for large-scene video and relates to the technical field of video processing.A social grouping method for large-scene video provided by the application is based on a large-scene social grouping framework of a space-time Transformer, and an end-to-end multi-person social relationship modeling method based on a graph model is proposed for a large-scene video with a large number of people; in view of the problems of occlusion and multi-modal information processing caused by the introduction of image information, an occlusion coding module and a space-time fusion Transformer module are designed on the basis of the multi-person social relationship modeling method based on the graph model, a time sequence information branch and a spatial image information branch are fused, the relationship between posture trajectory information and image information is better mined, and thus the social grouping result is more effectively improved.In a complex large-scene condition, the F1 index of the application is improved by more than 10% than the best existing method.

Description

Technical Field

[0001] This invention relates to the field of video processing technology, and in particular to a social grouping method for large-scene videos. Background Technology

[0002] Social grouping tasks aim to separate potentially socially active groups of people from images or videos. Due to limitations in image acquisition equipment, traditional social grouping methods are typically designed for small scenes with limited field of view, a moderate number of pedestrians, or closed scenes with limited types of object interactions and simple interaction patterns. This differs significantly from complex outdoor scenes. This significant difference limits the application of traditional social grouping methods, making them unsuitable for complex, open environments. In recent years, more and more researchers have shifted their focus to real-world, complex scenes, collecting corresponding image and video data. Real-world scenes offer a wider field of view, a larger number of pedestrians, richer interactions between individuals, and more complex group relationships; such scenes are defined as large-scale scenes.

[0003] As the world's most populous country, my country experiences crowd sizes and complexity in many public settings far exceeding those of existing research datasets, such as subway entrances, train stations, and large shopping malls. Monitoring and analyzing crowd movements in these scenarios can detect abnormal behavior, provide early warnings, and meet the application needs of public safety, smart cities, and other fields. Furthermore, in computer vision tasks centered on person analysis, such as group activity recognition and pedestrian trajectory prediction, people belonging to the same group may exhibit similar trajectories and behaviors, while people in different groups tend to maintain distance. Therefore, effective social grouping of crowds will help improve the performance of these downstream tasks. Thus, researching social grouping methods for large-scale video scenes has significant social and scientific value.

[0004] Traditional image-based social grouping methods are primarily designed for experimental datasets with limited participants, limited interactions, and simple visuals. Early, widely used image-based social grouping methods largely stemmed from the F-forms proposed by Kendon et al. (Kendon A. Conducting interaction: Patterns of behavior in focused encounters [M]. 1990.), which define F-forms as "forms in which two or more people form and remain together, engaging in conversation and exchanging information." Representative methods include Hough Voting (HVFF) (Cristani M, Bazzani L, Paggetti G, et al. Social interaction discovery by statistical analysis of F-formations [C]. In British Machine Vision Conference, 2014.) and Graph Clustering (DSFF) (Hung H, Kröse B. Detecting F-formations as dominant sets [C]. In Proceedings of the 13th International Conference on Multimodal Interfaces, 2011.). Some works (Swofford M, Peruzzi J, Tsoi N, et al. Improving social awareness through dante: Deep affinity network for clustering conversational interactants [J]. Proceedings of the ACM on Human-Computer Interaction, 2020,4 (CSCW1): 1–23.) employ graph neural networks to model interpersonal relationships. They utilize the location and pose information of each person in the graph as features to construct a fully connected graph to convey information between people. While these methods offer significant performance improvements over traditional methods, they neglect the crucial role of image information in social grouping, and the modeling approach of constructing fully connected graphs is impractical for large-scale datasets with thousands of participants.Recently, Li Kun et al. (Li Kun, Li Wanpeng, Sun Xiaokun et al. Deep social grouping networks for large-scale multi-object scenarios[J]. Science in China (Information Science), 2021, 51 (8): 1287–1301.) studied social grouping tasks based on large-scale scene images. They first used the original image information to capture the interaction behavior between people, and then used the depth information predicted from the original image to solve the problems of occlusion and false proximity in dense crowds. Although this method provides a new approach to extracting information from large-scale scene images, it also suffers from the common problems of previous methods: it does not utilize temporal information, and the predicted depth information is unreliable in unsupervised situations. In conclusion, while image-based social grouping methods can achieve good results in handling simple scenes, simply using location and orientation information is insufficient when facing real-world large-scale scenes. On the one hand, people's social positions change greatly in large-scale scenarios, and people may be far apart during social interactions, not always maintaining an "F-form," meaning that the "F-form" cannot encompass all forms of social interaction. On the other hand, in densely populated scenarios, everyone may be very close together, and the location information of individuals provided in a single image is simply not enough to effectively group people. More temporal and image information is needed to provide more basis for grouping.

[0005] With the iterative updates of hardware devices and the rapid development of machine learning, researchers have turned their attention to real-world outdoor scenes in recent years. Tsinghua University built a billion-pixel array camera and used this platform to collect the world's first billion-pixel dynamic large-scene multi-object dataset PANDA (Wang X, Zhang X, Zhu Y, et al. PANDA: Agigapixel-level human-centric video dataset [C]. In IEEE Conference on Computer Vision and Pattern Recognition, 2020.), filling the gap in large-scene, high-density crowd datasets. Currently, some researchers are also conducting research based on this dataset. Ehsanpour et al. (Ehsanpour M, Abedin A, Saleh F, et al. Joint learning of social groups, individuals action and sub-group activities in videos [C]. In European Conference on Computer Vision, 2020.) proposed a novel framework for processing small-scene videos and utilized image information. First, a classic video feature extraction network, I3D (Inflated 3D ConvNet), is used to obtain the temporal and spatial features of each person from multiple frames. Then, relationship modeling is performed for each pair of people in the video to construct a fully connected graph. Finally, graph partitioning is performed to obtain the corresponding social groups. This method works well for small scenes, but it is not suitable for high-resolution large scenes. On the one hand, common video feature extraction networks cannot handle ultra-high resolution, and on the other hand, large scene videos have a large number of people, making the construction of fully connected graphs inefficient. Wang et al. (Wang X, Zhang X, Zhu Y, et al. PANDA: Agigapixel-level human-centric video dataset [C]. In IEEE Conference on Computer Vision and Pattern Recognition, 2020.) proposed a global-to-local social grouping method for large scene videos. It takes multimodal information as input, uses trajectory information to coarsely screen the social relationships between people, and then introduces image information for further judgment for two people who may have a social relationship. Although the method takes into account the modeling of multi-person social relationships, the training process is two-stage and the inference process is separate. At the same time, the occlusion problem is ignored when introducing image features.Recently, Li et al. (Li J, Han R, Yan H, et al. Self-supervised social relation representation for human group detection [C]. In European Conference on Computer Vision, 2022.) proposed a self-supervised social grouping framework based on large-scene videos, which reduces the impact of mislabeling on the model. However, using pose information as input does not take into account the effect of occlusion. Therefore, traditional social grouping methods for limited viewpoints, a small number of pedestrians, or limited types of interactions between people are no longer suitable for large-scene videos with wide field of view, high resolution, and a large number of people. Recent social grouping methods for large-scene videos ignore some characteristics of large-scene videos and cannot provide more accurate and efficient grouping results.

[0006] To address the aforementioned issues, this invention proposes a social grouping method for large-scale video scenes. Summary of the Invention

[0007] The purpose of this invention is to propose a social grouping method for large-scene videos to address the problems mentioned in the background art. Based on a spatiotemporal Transformer-based large-scene social grouping framework, this invention addresses the problem of inefficiently modeling interpersonal social relationships in large-scene videos with numerous participants by designing an end-to-end multi-person social relationship modeling method based on a graph model. Building upon this multi-person social relationship modeling method, image information is introduced, and an occlusion coding structure and a Transformer network structure fusing multimodal information are designed to solve the problems of image occlusion and multimodal information fusion, respectively, effectively improving the results of social grouping for large-scene videos.

[0008] To achieve the above objectives, the present invention adopts the following technical solution:

[0009] A social grouping method for large-scene videos includes the following steps:

[0010] S1. Preprocess the large-scene video dataset, crop each person frame by frame from the original high-resolution video frame according to the labeled bounding box coordinates and scale it to a uniform size, extract image features and pose trajectory features for network training.

[0011] S2. Based on graph models, the complex social relationships between people are transformed into graph construction and graph analysis problems. In this stage, graph construction is carried out. According to the time strategy and spatial strategy, a sparse relationship graph between people is constructed to reduce the model's learning of invalid relationships.

[0012] S3. Input the low-level image features extracted in S1 into the occlusion coding module based on the self-attention mechanism to extract high-level image features;

[0013] S4. Input the low-level attitude trajectory information extracted in S2 into the temporal branch of the spatiotemporal Transformer to extract higher-level attitude trajectory features. Use the spatial branch in the spatiotemporal Transformer module to fuse it with the high-level image features obtained in S3, explore the relationship between the two, and generate vertex features.

[0014] S5. Input the sparse relation graph obtained in S2 and the vertex features obtained in S4 into the edge classification module for graph analysis.

[0015] S6. Based on the sparse relation graph obtained in S2 and the graph analysis results obtained in S5, the sparse relation graph is partitioned using a clustering method to obtain the final social grouping results.

[0016] Preferably, the preprocessing process described in S1 mainly includes the following steps:

[0017] S101. Based on the bounding box coordinates, first crop each person from the image and rescale them. size;

[0018] S102. Image feature extraction is performed using a pre-trained ResNet50, and the dimension of the last layer feature map of the model is increased from... Stretch into The image is input as low-level image features; 2D skeleton joint information is extracted frame-by-frame for each person using a Unipose network trained on the MPII dataset. It includes 16 key points, each with 2D coordinates in the image coordinate system. , These represent the person's serial number and the time serial number, respectively.

[0019] Preferably, the graph construction described in S2 mainly includes the following steps:

[0020] S201, Video Social Grouping Task: Input a video and multiple bounding box trajectories indicating pedestrian locations. The task is for the social grouping model to divide the people in the video into multiple groups based on their social relationships. Formally, the social grouping problem is a set partitioning problem. Let the set of pedestrians in the video be the complete set. Therefore, an ideal social grouping model needs to include the pedestrian set Divided into Non-intersecting groups That is, the following conditions must be met:

[0021]

[0022]

[0023]

[0024]

[0025] in, As an indicator function, when a pedestrian and pedestrians The function evaluates to 1 when a social relationship exists; otherwise, it evaluates to 0. Each person can only belong to one group, meaning they can belong to any two different groups. and There is no overlap between groups; people within the same group have social relationships with each other; people in different groups do not have social relationships. It is worth noting the number of groups. It is a variable that needs to be determined by the grouping algorithm;

[0026] S202. Based on the problem definition in S201, the social grouping problem can be based on indicator functions. Represented as an undirected weighted graph ,in, and Let them represent the set of vertices and the set of edges, respectively. This represents the edge weight mapping. Clearly, the set partitioning sought by social grouping can be applied to the graph. The graph is obtained by graph cutting. The graph construction method is based on manually constructed features from fully connected graphs. Generating subgraphs That is, to filter out as many as possible. Invalid edges in the model reduce depth. The computational load;

[0027] S203, The core of graph construction methods is to construct an edge filtering strategy. Filter out For invalid edges in the data, retain the set of edges of interest. Based on the task input, the appearance of each person in each frame can be obtained, including whether they appear and their location. Based on this, the present invention designs a time strategy and a space strategy respectively.

[0028] Timing Strategy: For any two people in a video, if they never appear in the same frame, then they cannot interact and cannot belong to the same group. Count the appearance of each person in T frames of video using a mask. This indicates that for any two people in the video... and Calculate the number of times they co-occur, that is, the number of times the two people appear together in the same frame of the video. :

[0029]

[0030] This indicates that the two people have never appeared in the same frame, and the invalid edge formed by these two people should be filtered out;

[0031] Spatial strategy: If two people appear simultaneously but always maintain a considerable distance, they cannot belong to the same group, and the edge formed by these two people is invalid. For two people who appear in the same frame, calculate their closest distance within a time interval T frames. :

[0032]

[0033]

[0034] in, Indicates the first Individuals in The position at any given moment. If Greater than a threshold This means that the two people always maintain a relatively large distance, and such invalid edges can also be discarded.

[0035] Combining time-based and space-based strategies, we can derive the edge filtering strategy:

[0036]

[0037] After the above filtering process, the graph construction method successfully transformed the initially fully connected graph, which was filled with invalid edges. It was transformed into a sparse relation graph. .

[0038] Preferably, the occlusion encoding module based on the self-attention mechanism described in S3 mainly includes the following steps:

[0039] S301, the occlusion encoding module mainly consists of transform... and Composition. The image features of the input consecutive frames are represented as... ,in , , These represent the number of people in the input, the dimension of the image features, and the number of video frames, respectively. Taking one person as an example, their consecutive frame image features are... express;

[0040] S302. Let's assume that this person is not occluded in most frames (this is relatively easy to ensure in long video data with large scenes). Then, for the same person, the appearance features are highly similar between unoccluded frames, while the features in the few occluded frames have lower similarity to other frames. Based on this characteristic, the similarity between image frames of the same person can be calculated by inner product in the normalized feature space, that is, the similarity between the frames of the same person in the th frame. Frame and the The similarity between frames can be represented as:

[0041]

[0042] The output is after passing through the ReLU activation layer. The similarity value ranges from 0 to 1.

[0043] S303, Current Number The average similarity between a frame and other frames reflects the occlusion level of that frame, i.e., the attention weight of the person in the current frame. If the first The more severe the frame occlusion, the lower its similarity to other frames, and consequently, The smaller the value, the greater the suppression of the frame's features. This can be expressed by the formula:

[0044]

[0045] S304. Pedestrians often don't appear in every frame of a video (e.g., they walk out of the frame), so in practice, calculating the mean similarity will ignore frames where these pedestrians aren't in the frame. Ultimately, attention weights are applied to the transformed... Based on the processed features, the output image features are obtained:

[0046]

[0047] S305, the occlusion coding module finally outputs the optimized image features for each person. .

[0048] Preferably, the spatiotemporal Transformer module described in S4 mainly includes the following steps:

[0049] S401. Based on the low-level pose trajectory information obtained in S102, a temporal branch is designed to extract high-level pose trajectory information. The temporal branch references the design structure within densely connected blocks in DenseNet. Except for the first layer, where the input is the original input, the input of each subsequent layer is the output features of all preceding layers, and the output of that layer also serves as the input to the next layer. This design improves feature reusability and simultaneously preserves the original pose and position features, as well as extracted high-level features such as velocity, acceleration, and motion. Since this invention only needs to process one-dimensional pose trajectory information, a 1D convolutional neural network is used instead of the previous 2D convolutional neural network to temporally fuse the skeleton information of each person frame by frame. Finally, the high-level pose trajectory information is obtained through the temporal branch. ;

[0050] S402. Utilize the encoder structure of the Transformer to learn the contextual information of an individual. For the input... Layer image features First, the attitude trajectory information is preprocessed. The features are concatenated to form the original embedded features of each individual. Note that the concatenated trajectory features can serve as positional encoding.

[0051]

[0052] S403. In the Transformer encoder, the temporal dimension is treated as a batch dimension, and spatial contextual information is extracted independently from all frames. The frame extraction process can be represented as:

[0053]

[0054]

[0055]

[0056] in, , , These are learnable parameters; MLP is a multilayer perceptron in the standard Transformer. It represents the features of all people at all times. It can be used This indicates that a spatiotemporal Transformer module ultimately outputs extracted features about two branches, namely... and This can be used as the feature input for the next spatiotemporal Transformer module. Stacking the spatiotemporal Transformer modules can form a deep model.

[0057] S404. Image features and pose trajectory features output by spatiotemporal Transformer modules at different depths are concatenated to form vertex features, expressed by the formula:

[0058]

[0059] Note that the above process preserves the time dimension information.

[0060] Preferably, the graph analysis process described in S5 mainly includes the following steps:

[0061] S501. Based on the vertex features obtained in S404, construct edge features to predict edge weights. (Consider human...) and exist Social relationships at any given moment, i.e., edge Construct edge features using the difference between point features between two people. Expressed as a formula:

[0062]

[0063] in, This represents taking the absolute value of each element. Considering that an undirected graph is being constructed, the edge features are designed using the symmetry of the absolute value operation, i.e. This means that differences in the input order do not change the edge features.

[0064] S502, Edge Features The data is input into a fully connected layer (MLP) for frame-by-frame classification, and finally, mean pooling is used in the temporal dimension to obtain the final relationship score, expressed by the formula:

[0065]

[0066]

[0067] Next, predict the edge weights. ,in This refers to the Sigmod function;

[0068] S503. Based on the labeled grouping information, each edge corresponds to a label. , Represents people and people They belong to the same group, obviously, assuming the labeling is correct. Finally, the model is trained using binary cross-entropy loss.

[0069] Preferably, the binary cross-entropy loss described in S503 is expressed by the formula:

[0070]

[0071] Preferably, the clustering method described in S6 mainly includes the following steps:

[0072] S601. Based on the edge weight prediction results obtained in S502 and the sparse relation graph obtained in S203. Using a label propagation strategy, edge connections are iteratively deleted or merged based on edge weight prediction results;

[0073] S602. As the number of edges in the graph decreases... The data will be divided into disjoint subgraphs, which will serve as the final social grouping results for the population.

[0074] Compared with existing technologies, this invention provides a social grouping method for large-scene videos, which has the following beneficial effects:

[0075] (1) This invention proposes a social grouping method for large-scene videos, which can achieve high-efficiency social grouping; at the same time, it proposes an end-to-end multi-person social relationship modeling method based on graph model, which can improve the efficiency of multi-person social relationship modeling; it also proposes an occlusion coding method based on self-attention mechanism, which can efficiently and accurately extract personal features; finally, it also proposes a spatiotemporal fusion Transformer method, which can fuse multimodal information.

[0076] (2) This invention proposes an end-to-end multi-person social relationship modeling method based on graph models, which solves the problem of inefficient modeling when there are many people in large-scale video scenes. It transforms the complex modeling of interpersonal social relationships into a graph representation and graph analysis problem, which reduces the model's learning of invalid relationships and improves the model's ability to model large numbers of people. It also enables end-to-end training of the model.

[0077] (3) This invention proposes an occlusion coding method based on self-attention mechanism to solve the occlusion problem caused by the introduction of image information. The average similarity between the current frame and other frames reflects the occlusion status of the frame, which suppresses the features of severely occluded frames and improves the robustness to occlusion.

[0078] (4) This invention proposes a spatiotemporal fusion Transformer method to solve the problem of multimodal information fusion caused by the introduction of image information. The spatiotemporal fusion Transformer fuses the attitude trajectory information extracted by the temporal branch and the image information extracted by the spatial branch, and mines more important feature information from the multimodal information for social relationship grouping.

[0079] (5) Experimental results on the large-scene video dataset PANDA fully demonstrate the effectiveness of the present invention in social grouping tasks. Compared with the best current method, the present invention improves the F1 score by more than 10%. Attached Figure Description

[0080] Figure 1 This is a schematic diagram of the spatiotemporal Transformer-based social grouping framework for large-scene videos proposed in this invention.

[0081] Figure 2 This is a schematic diagram of the social grouping results on the PANDA dataset in the social grouping method for large-scene videos proposed in this invention;

[0082] Figure 3 This is a qualitative comparison chart of the proposed social grouping method for large-scene videos and mainstream social grouping methods in the prior art. Detailed Implementation

[0083] The technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments.

[0084] To address the limitations of existing social grouping methods for large-scale video scenes in achieving satisfactory results and the challenges of handling large-scale video datasets, this invention proposes a spatiotemporal Transformer-based framework for large-scale social grouping. For the large number of participants in large-scale videos, an end-to-end multi-person social relationship modeling method based on a graph model is proposed. This transforms the complex modeling of interpersonal social relationships into a graph representation and analysis problem, reducing the model's learning of invalid relationships and improving its ability to model large numbers of participants. It also enables end-to-end training of the model. To address the occlusion problem introduced by image features, an occlusion encoding method based on a self-attention mechanism is proposed to extract individual features. The average similarity between the current frame and other frames reflects the occlusion status of the frame, suppressing features in severely occluded frames. To address the multimodal information fusion problem introduced by image features, a spatiotemporal fusion Transformer method is proposed. This method fuses the pose trajectory information extracted from the temporal branch and the image information extracted from the spatial branch, mining more important feature information from the multimodal information for social relationship grouping. Specific examples are as follows.

[0085] Example 1:

[0086] Please see Figure 1 This invention proposes a method for acquiring and processing real-world raw video denoising datasets based on screen images, comprising the following steps:

[0087] S1. Preprocess the large-scene video dataset, crop each person frame by frame from the original high-resolution video frame according to the labeled bounding box coordinates and scale it to a uniform size, extract image features and pose trajectory features for network training.

[0088] The preprocessing process described in S1 mainly includes the following steps:

[0089] S101. Based on the bounding box coordinates, first crop each person from the image and rescale them. size;

[0090] S102. Image feature extraction is performed using a pre-trained ResNet50, and the dimension of the last layer feature map of the model is increased from... Stretch into The image is input as low-level image features; 2D skeleton joint information is extracted frame-by-frame for each person using a Unipose network trained on the MPII dataset. It includes 16 key points, each with 2D coordinates in the image coordinate system. , These represent the person's serial number and the time serial number, respectively.

[0091] S2. Based on graph models, the complex social relationships between people are transformed into graph construction and graph analysis problems. In this stage, graph construction is carried out. According to the time strategy and spatial strategy, a sparse relationship graph between people is constructed to reduce the model's learning of invalid relationships.

[0092] The graph construction described in S2 mainly includes the following steps:

[0093] S201, Video Social Grouping Task: Input a video and multiple bounding box trajectories indicating pedestrian locations. The task is for the social grouping model to divide the people in the video into multiple groups based on their social relationships. Formally, the social grouping problem is a set partitioning problem. Let the set of pedestrians in the video be the complete set. Therefore, an ideal social grouping model needs to include the pedestrian set Divided into Non-intersecting groups That is, the following conditions must be met:

[0094]

[0095]

[0096]

[0097]

[0098] in, As an indicator function, when a pedestrian and pedestrians The function evaluates to 1 when a social relationship exists; otherwise, it evaluates to 0. Each person can only belong to one group, meaning they can belong to any two different groups. and There is no overlap between groups; people within the same group have social relationships with each other; people in different groups do not have social relationships. It is worth noting the number of groups. It is a variable that needs to be determined by the grouping algorithm;

[0099] S202. Based on the problem definition in S201, the social grouping problem can be based on indicator functions. Represented as an undirected weighted graph ,in, and Let them represent the set of vertices and the set of edges, respectively. This represents the edge weight mapping. Clearly, the set partitioning sought by social grouping can be applied to the graph. The graph is obtained by graph cutting. The graph construction method is based on manually constructed features from fully connected graphs. Generating subgraphs That is, to filter out as many as possible. Invalid edges in the model reduce depth. The computational load;

[0100] S203, The core of graph construction methods is to construct an edge filtering strategy. Filter out For invalid edges in the data, retain the set of edges of interest. Based on the task input, the appearance of each person in each frame can be obtained, including whether they appear and their location. Based on this, the present invention designs a time strategy and a space strategy respectively.

[0101] Timing Strategy: For any two people in a video, if they never appear in the same frame, then they cannot interact and cannot belong to the same group. Count the appearance of each person in T frames of video using a mask. This indicates that for any two people in the video... and Calculate the number of times they co-occur, that is, the number of times the two people appear together in the same frame of the video. :

[0102]

[0103] This indicates that the two people have never appeared in the same frame, and the invalid edge formed by these two people should be filtered out;

[0104] Spatial strategy: If two people appear simultaneously but always maintain a considerable distance, they cannot belong to the same group, and the edge formed by these two people is invalid. For two people who appear in the same frame, calculate their closest distance within a time interval T frames. :

[0105]

[0106]

[0107] in, Indicates the first Individuals in The position at any given moment. If Greater than a threshold This means that the two people always maintain a relatively large distance, and such invalid edges can also be discarded.

[0108] Combining time and space strategies, we can obtain the edge filtering strategy S:

[0109]

[0110] After the above filtering process, the graph construction method successfully transformed the initially fully connected graph, which was filled with invalid edges. It was transformed into a sparse relation graph. .

[0111] S3. Input the low-level image features extracted in S1 into the occlusion coding module based on the self-attention mechanism to extract high-level image features;

[0112] The occlusion coding module based on the self-attention mechanism described in S3 mainly includes the following steps:

[0113] S301, the occlusion encoding module mainly consists of transform... and Composition. The image features of the input consecutive frames are represented as... ,in , , These represent the number of people in the input, the dimension of the image features, and the number of video frames, respectively. Taking one person as an example, their consecutive frame image features are... express;

[0114] S302. Let's assume that this person is not occluded in most frames (this is relatively easy to ensure in long video data with large scenes). Then, for the same person, the appearance features are highly similar between unoccluded frames, while the features in the few occluded frames have lower similarity to other frames. Based on this characteristic, the similarity between image frames of the same person can be calculated by inner product in the normalized feature space, that is, the similarity between the frames of the same person in the th frame. Frame and the The similarity between frames can be represented as:

[0115]

[0116] The output is after passing through the ReLU activation layer. The similarity value ranges from 0 to 1.

[0117] S303, Current Number The average similarity between a frame and other frames reflects the occlusion level of that frame, i.e., the attention weight of the person in the current frame. If the first The more severe the frame occlusion, the lower its similarity to other frames, and consequently, The smaller the value, the greater the suppression of the frame's features. This can be expressed by the formula:

[0118]

[0119] S304. Pedestrians often don't appear in every frame of a video (e.g., they walk out of the frame), so in practice, calculating the mean similarity will ignore frames where these pedestrians aren't in the frame. Ultimately, attention weights are applied to the transformed... Based on the processed features, the output image features are obtained:

[0120]

[0121] S305, the occlusion coding module finally outputs the optimized image features for each person. .

[0122] S4. Input the low-level attitude trajectory information extracted in S2 into the temporal branch of the spatiotemporal Transformer to extract higher-level attitude trajectory features. Use the spatial branch in the spatiotemporal Transformer module to fuse it with the high-level image features obtained in S3, explore the relationship between the two, and generate vertex features.

[0123] The spatiotemporal Transformer module described in S4 mainly includes the following steps:

[0124] S401. Based on the low-level pose trajectory information obtained in S102, a temporal branch is designed to extract high-level pose trajectory information. The temporal branch references the design structure within densely connected blocks in DenseNet. Except for the first layer, where the input is the original input, the input of each subsequent layer is the output features of all preceding layers, and the output of that layer also serves as the input to the next layer. This design improves feature reusability and simultaneously preserves the original pose and position features, as well as extracted high-level features such as velocity, acceleration, and motion. Since this invention only needs to process one-dimensional pose trajectory information, a 1D convolutional neural network is used instead of the previous 2D convolutional neural network to temporally fuse the skeleton information of each person frame by frame. Finally, the high-level pose trajectory information is obtained through the temporal branch. ;

[0125] S402. Utilize the encoder structure of the Transformer to learn the contextual information of an individual. For the input... Layer image features First, the attitude trajectory information is preprocessed. The features are concatenated to form the original embedded features of each individual. Note that the concatenated trajectory features can serve as positional encoding.

[0126]

[0127] S403. In the Transformer encoder, the temporal dimension is treated as a batch dimension, and spatial contextual information is extracted independently from all frames. The frame extraction process can be represented as:

[0128]

[0129]

[0130]

[0131] in, , , These are learnable parameters; MLP is a multilayer perceptron in the standard Transformer. It represents the features of all people at all times. It can be used This indicates that a spatiotemporal Transformer module ultimately outputs extracted features about two branches, namely... and This can be used as the feature input for the next spatiotemporal Transformer module. Stacking the spatiotemporal Transformer modules can form a deep model.

[0132] S404. Image features and pose trajectory features output by spatiotemporal Transformer modules at different depths are concatenated to form vertex features, expressed by the formula:

[0133]

[0134] Note that the above process preserves the time dimension information.

[0135] S5. Input the sparse relation graph obtained in S2 and the vertex features obtained in S4 into the edge classification module for graph analysis.

[0136] The graph analysis process described in S5 mainly includes the following steps:

[0137] S501. Based on the vertex features obtained in S404, construct edge features to predict edge weights. (Consider human...) and exist Social relationships at any given moment, i.e., edge Construct edge features using the difference between point features between two people. Expressed as a formula:

[0138]

[0139] in, This represents taking the absolute value of each element. Considering that an undirected graph is being constructed, the edge features are designed using the symmetry of the absolute value operation, i.e. This means that differences in the input order do not change the edge features.

[0140] S502, Edge Features The data is input into a fully connected layer (MLP) for frame-by-frame classification, and finally, mean pooling is used in the temporal dimension to obtain the final relationship score, expressed by the formula:

[0141]

[0142]

[0143] Next, predict the edge weights. ,in This refers to the Sigmod function;

[0144] S503. Based on the labeled grouping information, each edge corresponds to a label. , Represents people and people They belong to the same group, obviously, assuming the labeling is correct. Finally, the model is trained using binary cross-entropy loss:

[0145]

[0146] S6. Based on the sparse relation graph obtained in S2 and the graph analysis results obtained in S5, the sparse relation graph is partitioned using a clustering method to obtain the final social grouping results.

[0147] The clustering method described in S6 mainly includes the following steps:

[0148] S601. Based on the edge weight prediction results obtained in S502 and the sparse relation graph obtained in S203. Using a label propagation strategy, edge connections are iteratively deleted or merged based on edge weight prediction results;

[0149] S602. As the number of edges in the graph decreases... The data will be divided into disjoint subgraphs, which will serve as the final social grouping results for the population.

[0150] Example 2:

[0151] Please see Figure 1-3 Based on Example 1, but with the following differences:

[0152] The specific implementation process is as follows:

[0153] (a) Data preprocessing:

[0154] This invention utilizes PANDA, the world's first large-scene dataset, in the training phase.

[0155] Scenes 02 OCT Habour, 03 Xili Crossroad, 04 Primary School, 05 Basketball Court, 06 Xinzhongguan, 07 University Campus, 08 Xili Street 1, and 09 Xili Street 2 from the PANDA dataset were selected, with scene 01 University Canteen chosen for the test portion. Each person was first cropped from the video frames based on their bounding box coordinates and then resized to the same size. Image features were extracted using a pre-trained ResNet50 as low-level image feature input. Then, a Unipose network trained on the MPII dataset was used to extract 2D skeleton joint information for each person frame-by-frame as low-level pose trajectory feature input.

[0156] (ii) Graph construction:

[0157] Based on the characteristics of handcrafted construction, a time-based and space-based strategy was designed to initially filter out unsuitable materials.

[0158] Edges that may contain social relationships can be reduced by filtering the fully connected graph into a sparse graph, thereby improving the efficiency of learning edge relationships.

[0159] (III) Occlusion Encoding Module:

[0160] The low-level image features obtained in step (i) are input into the occlusion coding module for further feature extraction. For the same person, when appearing in different frames without occlusion, the extracted features of that person in these frames should be highly similar, while the features in occluded frames have very low similarity. Specifically, the image information of different frames is first input into the occlusion coding method. The occlusion attention weight of each frame can be defined as the average similarity with all other frames. The occlusion attention weight is used to reduce the impact of severely occluded frames on the final personal feature representation, thereby improving the robustness of the invention to the occlusion problem.

[0161] (iv) Spatiotemporal Transformer Module:

[0162] The low-level pose feature trajectory obtained in step (1) is input into the spatiotemporal Transformer.

[0163] The temporal branch extracts higher-level pose trajectory features. The temporal branch is mainly designed based on the 1D convolutional reference DenseNet structure. The high-level image features and high-level pose trajectory features obtained in step (3) are concatenated and input into the spatial branch of the spatiotemporal Transformer to form individual original embedded feature inputs, explore the relationship between multimodal features, and provide more grouping criteria. By stacking the number of spatiotemporal Transformer modules, a deep network structure can be formed. The image features and pose trajectory features output by spatiotemporal Transformer modules of different depths are concatenated to form vertex features.

[0164] (V) Graph Analysis:

[0165] The sparse relation graph obtained in step (II) and the vertex features obtained in step (IV) are input into the edge classification module for graph analysis. Edge features between people are constructed based on the vertex features, and then the edge features are weighted and evaluated to obtain a score of the closeness between people. Finally, based on the edge feature weights, a clustering algorithm is used to partition the sparse relation graph, obtaining the final social grouping results.

[0166] like Figure 1The diagram illustrates the large-scene social grouping framework based on spatiotemporal Transformer proposed in this invention. Large-scene images are cropped based on existing bounding box annotations, and preprocessed to obtain image features and pose trajectory features as important inputs to the network. Using handcrafted features, the fully connected relationship graph between people is analyzed and filtered to obtain a sparse relationship graph. An occlusion encoding module is designed to address the occlusion problem introduced by image features. A spatiotemporal Transformer is designed to mine the relationships between multimodal feature inputs (image features and pose trajectory features) to form vertex features. Finally, edge weight analysis is performed on the sparse relationship graph using vertex features, and a clustering algorithm is used to partition the sparse relationship graph, achieving more efficient and accurate social grouping.

[0167] like Figure 2 As shown, the social grouping results of the present invention on the PANDA dataset are presented. The results fully demonstrate that the large-scene social grouping method proposed in this invention has a certain social grouping capability and can obtain satisfactory social grouping results in non-extreme scenarios.

[0168] like Figure 3 As shown, the qualitative results of this invention are compared with those of the current mainstream large-scene video social grouping methods. It can be seen that this method can divide reasonable social grouping results in more complex situations.

[0169] Table 1 lists the quantitative results of this invention and current mainstream social grouping methods for large-scene videos on the PANDA dataset; Global, Global-to-local w Random, and Global-to-local wUncertainty are respectively the results of Wang et al. (Wang X, Zhang X, Zhu Y, et al PANDA: A gigapixel-level human-centric video dataset [C]. In IEEE Conference on Computer Vision and Pattern Recognition, 2020. Three baseline methods were proposed in 2020; S3R2 is a method proposed by Li et al. (Li J, Han R, Yan H, et alThe self-supervised method, proposed in 2022 (Self-supervised social relation representation for humangroup detection [C]. In European Conference on Computer Vision, 2022.), uses Precision, Recall, and F1 as evaluation metrics for quantitative results (see Table 1).

[0170] Table 1

[0171]

[0172] As shown in Table 1, the experimental results on the large-scene video dataset PANDA fully demonstrate the effectiveness of this invention in social grouping tasks. Compared to the best current method, this invention improves the F1 score by more than 10%.

[0173] The above description is only a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any equivalent substitutions or modifications made by those skilled in the art within the scope of the technology disclosed in the present invention, based on the technical solution and inventive concept of the present invention, should be covered within the scope of protection of the present invention.

Claims

1. A social grouping method for large scene video, characterized in that, Specifically comprising the following steps: S1, preprocessing the large scene video dataset, according to the annotated bounding box coordinates, each person is cropped from the high-resolution original video frame by frame and scaled to a uniform size, low-level image features and pose trajectory features are extracted for network training; S2, based on the graph model, the complex social relationship between people is modeled into a graph construction and graph analysis problem, in the graph construction process, according to the time strategy and the space strategy, the sparse relationship graph between people is constructed, and the learning of invalid relationship by the model is reduced; S3, the low-level image features obtained in S1 are input into the occlusion coding module based on the self-attention mechanism to extract high-level image features; S4, the low-level pose trajectory information extracted in S1 is input into the time sequence branch of the space-time Transformer to extract higher-level pose trajectory features, and the space branch in the space-time Transformer module is used to fuse the high-level image features obtained in S3, the relationship between the two is mined, and vertex features are generated; S5, the sparse relationship graph obtained in S2 and the vertex features obtained in S4 are input into the edge classification module for graph analysis; S6, according to the sparse relationship graph obtained in S2 and the graph analysis result obtained in S5, the sparse relationship graph is divided by clustering method to obtain the final social grouping result.

2. The social grouping method for large scene video of claim 1, wherein, The data set preprocessing process in S1 specifically comprises the following steps: S101, according to the bounding box coordinates, each person is cropped from the image and rescaled to 128x64 size; S102, using a pre-trained ResNet50 to extract image features, stretching the last layer feature map of the model from 2048x4x2 to 16384 dimensions as low-level image features input; using the Unipose network trained on the MPII dataset to extract 2D skeleton joint information for each person frame by frame, denoted as ; the 2D skeleton joint includes 16 nodes, and the coordinates of each node are 2D coordinates in the image coordinate system, wherein, i , t respectively represent the person number and the time number.

3. The social grouping method for large scene video of claim 1, wherein, The graph construction in S2 specifically comprises the following steps: S201, the video social grouping task inputs a video and a plurality of bounding box trajectories identifying the positions of pedestrians, and uses a social grouping model to divide the people in the video into a plurality of groups according to social relationships, and converts the social grouping problem into a set partitioning problem, which specifically includes the following contents: Let the set of all pedestrians in the video be Then the ideal social grouping model needs to divide the set of pedestrians into mutually disjoint groups that satisfy the following conditions: wherein, is an indicator function, which equals 1 if the pedestrian and the pedestrian have a social relationship, and 0 otherwise; each person can only belong to one of the groups, and there is no intersection between any two different groups and ; people in the same group have a social relationship with each other; people in different groups do not have a social relationship with each other; k denotes the number of groups, which is a variable determined by the grouping algorithm; S202, define the social grouping problem based on the indicator function defined in S201 is represented as an undirected weighted graph wherein, and represent vertex set and edge set respectively, represents edge weight mapping; the graph is cut to obtain the set partitioning sought by social grouping; the graph construction method is based on manually constructed features from the full connection graph generates sub-graphs , filters out invalid edges in , and reduces the computational load of the deep model . S203, construct edge filtering strategy S , filter out invalid edges in , and keep the edge set of interest ; based on the task input, get the appearance of each person in each frame, including whether to appear, position information, and design time strategy and space strategy.

4. The social grouping method for large scene video of claim 3, wherein, The time strategy and the space strategy specifically include the following contents: Time strategy: for any two people in the video, if they never appear in the same frame of video, they do not exist interaction, also not belong to the same group; count the appearance of each person in T frame of video, use mask to represent, for any two people and in the video, calculate the co-occurrence between them, that is, the number of times that two people appear in the same frame of video , the specific calculation formula is: If , it means that the two persons never appeared in the same frame, and the invalid edge formed by the two persons is filtered out. Spatial strategy: two people appear at the same time but always keep a distance, they do not belong to the same group, the edge formed by the two people is invalid edge; for two people appearing in the same frame, calculate the nearest distance between them in T frame time , the specific calculation formula is: wherein, represents the person at a moment in time; if is greater than a threshold , it indicates that the two persons always keep a large distance, and the invalid edge is discarded; Combining the above time strategy and space strategy, the edge filtering strategy is obtained S : After the above filtering is done, realize the full connection graph with invalid edges Convert to sparse relation graph .

5. The social grouping method for large scene video of claim 1, wherein, The S3 specifically includes the following contents: S301, the occlusion encoding module based on self-attention mechanism is composed of transformation and ; the image feature representation of the input continuous frame is represented as , , , respectively represent the number of input people, the dimension of image feature and the number of video frames; S302, assuming that the selected person is not occluded in most frames, for the same person, the appearance features between the un-occluded frames are highly similar, and the features of the few occluded frames are less similar to other frames, according to the characteristics, the similarity between the image frames of the same person is calculated by inner product in the normalized feature space, and the similarity between the person in the i-th frame and the person in the j-th frame is represented as: Sij= (f i - f j)T (f i - f j) Sij= (f i - f j)T (f i - f j) After the ReLU activation layer, output, The value range of the similarity is between 0 and 1. S303、current the The average similarity of the current frame and other frames reflects the occlusion condition of the current frame, indicating the attention weight of the person in the current frame ; If the frame is the first frame The more serious the occlusion case is, the lower the similarity of the frame with other frames is, and correspondingly, The value of is also smaller, and the degree of suppression of the characteristics of the frame is greater, which is specifically expressed as: S304. Pedestrians will not appear in every frame of the video. In actual implementation, frames where pedestrians are not in the frame are ignored when calculating the mean similarity. Finally, attention weights are applied to the transformed... Based on the processed features, the output image features are obtained: S305, the occlusion coding module finally outputs the optimized image features of each person .

6. The social grouping method for large scene video of claim 1, wherein: The S4 specifically includes the following contents: S401, according to the low-level posture trajectory information obtained in S1, a timing branch is designed to extract high-level posture trajectory information; the timing branch refers to the design structure inside the dense connection block in DenseNet, except that the input of the first layer is the original input, the input of each layer is the output features of all previous layers, and the output of this layer is also used as the input of the next layer; a 1D convolutional neural network is used instead of the previous 2D convolutional neural network to fuse the frame-by-frame skeleton information of each person in time sequence; finally, high-level posture trajectory information is obtained through the timing branch ; S402, learn the context information of the individual by using the encoder structure of the Transformer; for the input of the first layer image features , first and pre-processed pose trajectory information are spliced to form the original embedding feature input of the individual, and the spliced trajectory features serve as position encoding: S403. In the Transformer encoder, the temporal dimension is treated as a batch dimension to independently extract the context information in space from all frames; the first The extraction process of the frame is represented as: wherein, , , are learnable parameters, MLP is a multi-layer perceptron in standard Transformer; the features of all people at all time points are denoted as ; a spatio-temporal Transformer module finally outputs extracted features about two branches, which are and , respectively, as the feature input of the next spatio-temporal Transformer module; the number of stacked spatio-temporal Transformer modules forms a deep model;​ S404, the image features and the pose trajectory features output by the space-time Transformer module at different depths are spliced into vertex features, and the specific formula is: The splicing process retains the time dimension information.

7. The social grouping method for large scene video of claim 1, wherein: The S5 specifically includes the following contents: S501、According to the vertex features obtained in S4, construct edge features to predict edge weights; consider the human and At the moment of social relationship, edge , construct edge features using the difference between the point features of two people , and the specific formula is as follows: wherein represents taking the absolute value element-wise; S502、the edge feature Input into the full connection layer (MLP) frame by frame classification, and the final relationship score is obtained through mean pooling in the time dimension, and the specific formula is represented as: The obtained relationship score is used to predict the edge weight: wherein represents a Sigmod function; S503、According to the labeled grouping information, all edges correspond to a label , representing a person and a person belongs to the same group, and in the case of correct labeling ; train the model using binary cross-entropy loss.

8. The social grouping method for large scene video according to claim 7, characterized in that, The binary classification cross-entropy loss in S503 is specifically represented by the formula: Wherein, λ represents the balance coefficient, and the specific formula is: wherein represents the set of valid edges.

9. The social grouping method for large scene video of claim 1, wherein: The S6 specifically includes the following contents: S601, according to the edge weight prediction result obtained in S5 and the sparse relationship graph obtained in S2, a label propagation strategy is used to iteratively delete or merge the edge connection relationship according to the edge weight prediction result; S602, as the number of edges in the graph decreases, the sparse relationship graph is divided into disconnected subgraphs, and the subgraphs are taken as the final social grouping result of the crowd.

Images