A self-supervised group detection method and device for large-scale multi-object detection

Abstract

The present invention discloses a self-supervised group detection method and device for large-scale scenes and multiple objects. The method includes: modeling N people in each t-frame video of a crowded multi-person scene; extracting feature information such as activity and time trajectory for each person; training an environmental perception human behavior simulator in a self-supervised manner, that is, spontaneously learning natural human behavior in a group; based on the idea of ​​causal reasoning, using the environmental perception human behavior simulator to train a relationship network to discover paired interpersonal relationships between multiple people; determining the parameters of the relationship network, and adding a network based on a fully connected layer to realize the task of artificial group detection. The device includes: a processor and a memory. Based on the idea of ​​causal reasoning, the present invention can spontaneously learn natural human behavior in a group without the need for a large amount of labeled information, thereby completing a better group detection task.

Description

Technical Field

[0001] The present invention relates to the field of multi-person scene group detection, and in particular to a self-supervised group detection method and device for large-scale scene multi-object detection. Background Art

[0002] Video surveillance is currently widely used, and group relationships within video surveillance are a crucial component of video content analysis. Group detection involves segmenting a large group of people into numerous smaller, non-overlapping groups. This is crucial for video-based analysis of human social activities and forms the basis for analyzing group behavior. The difficulties and challenges of this task lie in the following: First, compared to detecting a single person, the composition of a group is significantly more complex, making detection inherently challenging. Second, the sheer number of people and groups in large-scale group detection datasets places high demands on model performance. Furthermore, the workload and difficulty of labeling datasets is substantial, further limiting training for group detection tasks.

[0003] In existing methods, crowd detection first uses pedestrian trajectories or local spatial features to extract features, and then completes crowd detection through similarity measurement and cluster analysis. This approach has two problems: First, most previous crowd detection features are predefined, and in fact many are manually set, including time trajectories and spatial distances. This not only limits the accuracy of the model but also makes it difficult to detect crowds in very large scenes. Second, previous methods generally require the use of fully supervised annotation, which is very large-scale and therefore time-consuming and labor-intensive. Summary of the Invention

[0004] This invention provides a self-supervised group detection method and device for large-scale multi-object scenarios. Based on the idea of ​​causal reasoning, the invention can spontaneously learn natural human behavior in a group without requiring a large amount of labeled information, thereby achieving better group detection tasks. The details are described below:

[0005] In a first aspect, a self-supervised group detection method for multiple objects in a large scene is provided, the method comprising:

[0006] Model N people in every t frames of a crowded multi-person scene video; extract feature information such as activity and time trajectory for each person;

[0007] Train an environment-aware human behavior simulator in a self-supervised manner, i.e., spontaneously learn natural human behaviors in a group;

[0008] Based on the idea of ​​causal reasoning, the relationship network is trained using an environmental perception human behavior simulator to discover pairwise interpersonal relationships between multiple people. The parameters of the relationship network are determined, and a network based on a fully connected layer is added to realize the artificial group detection task.

[0009] Among them, a self-supervised training of an environment-aware human behavior simulator, that is, spontaneously learning natural human behavior in a group, is as follows:

[0010] Destroyer The state of is changed by space-time conversion to obtain Will be input into the restorer network and restored to The recovery network is represented as RecNet, and the training process is expressed as:

[0011]

[0012] in, It is the central subject feature after time-space transformation; It is a characteristic cluster formed by combining the central subject and the surrounding subjects after time-space transformation;

[0013] in, Represents the c main feature, Represents the characteristics of the n neighbors of the c subject, and defines is a feature cluster combining the central subject and the surrounding subjects, D represents an implicit distribution to be learned, n represents the set of n nearby neighbors, Nc represents the set of all objects in the entire scene, and T represents the total number of frames in the video.

[0014] The relationship network is trained based on the idea of ​​causal reasoning and using an environmental perception human behavior simulator to discover pairwise interpersonal relationships between multiple people. The parameters of the relationship network are specifically determined as follows:

[0015] Given a coded feature at time t The subject i, the neighbor set N i Contains N-1 subjects, and the relationship between subject i and all subjects is represented as a probability vector in For the target person and its neighbor j, the relationship probability is calculated through several steps As shown below:

[0016] Delete the feature embedding of subject j, i.e. v j and Figure G t The edge involving subject j in the current feature cluster is represented as middle The feature embedding representing subject j is eliminated from the surrounding subjects;

[0017] The proposed behavior simulator is used to recover the tampering signature cluster, i.e., From this we get the recovered features of subject i Calculate the relationship probability vector of j's influence on i until all neighbors of i are calculated, and use the Softmax function to convert Normalized to meet And get the relationship vector of i

[0018] The goal is to learn a human relationship extraction network from feature groups The probability of learning the relationship is:

[0019]

[0020] in, Represents the predicted relation vector from RelNet, and finally determines the parameters of the relation network.

[0021] In a second aspect, a self-supervised group detection device for large-scale scenes and multiple objects is provided, the device comprising: a processor and a memory, the memory storing program instructions, the processor calling the program instructions stored in the memory to enable the device to execute any one of the method steps described in the first aspect.

[0022] The beneficial effects of the technical solution provided by the present invention are:

[0023] 1. Based on the idea of ​​causal reasoning, this invention can spontaneously learn natural human behavior in a group without requiring a large amount of labeled information. This can solve group detection tasks in complex multi-person scenarios, thus providing a prerequisite for analyzing group behavior status.

[0024] 2. This paper constructs a large-scale virtual group detection dataset with rich annotations. Experimental results on the new virtual dataset and another latest benchmark dataset PANDA verify the effectiveness of this method, with high group detection accuracy;

[0025] 3. This paper establishes a self-supervised, context-aware human behavior simulator that can derive bidirectional interpersonal relationships between characters. For example, for a target subject 'a', if the elimination of surrounding person 'b' has little effect on 'a's' recovery, the relationship between 'a' and 'b' is considered weak; otherwise, the significant influence of 'c' leads to a strong relationship. This allows the simulation of relevant interpersonal relationships.

[0026] 4. This paper builds a two-stage trainable crowd detection model based on the "environmental perception human behavior simulator", which has very good crowd detection effect;

[0027] 5. Self-supervised learning: This invention does not require a large amount of data annotation for training, has strong versatility, and is highly adaptable to the environment. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] Figure 1 This is a flowchart of a self-supervised group detection method for multiple objects in large scenes;

[0029] Figure 2 This is a structural diagram of a self-supervised group detection device for large scenes and multiple objects. DETAILED DESCRIPTION

[0030] In order to make the objectives, technical solutions and advantages of the present invention more clear, the embodiments of the present invention are described in further detail below. Technical background:

[0032] (1) Modeling of interpersonal relationship graphs (see reference [1]): In multi-person scenarios, modeling of relationships between roles is of great significance for identifying group activities. Using deep models to effectively learn the discriminative relationships between actors has been widely used in multi-person scenario group recognition and behavior recognition. To this end, the embodiment of the present invention constructs a flexible and efficient participant relationship graph to simultaneously capture the appearance and position relationships between participants. This approach can flexibly and effectively explain the relationships between participants, while capturing the appearance and position relationships between participants for group activity recognition. It provides an interpretable mechanism that can explicitly model the relevant relationships between people in the scene, thereby improving the ability to identify different group activities.

[0033] (2) Social-STGCNN (see reference [2]): The trajectory of a pedestrian is not only affected by the pedestrian himself, but also by the interaction with the surrounding objects. Previous methods model these interactions by using various aggregation methods that integrate different learner pedestrian states. The embodiment of the present invention utilizes a social spatiotemporal graph convolutional neural network (Social-STGCNN), which replaces the need for aggregation methods by modeling interactions as graphs. STGCNN is a spatiotemporal graph CNN that was originally designed to solve the problem of skeleton-based action recognition. In the work of the embodiment of the present invention, ST-GCNN extracts spatial and temporal information from the graph to create a suitable embedding.

[0034] (3) Causal reasoning (see reference [3]): Causal reasoning is an important research topic involving many fields such as statistics, computer science, education, public policy, and economics. Currently, using observational data to estimate causal effects has become an attractive research direction due to the large amount of data and low budget requirements compared to randomized controlled trials. Causal relationships are also called causal effects, in which the cause is partially responsible for the result, and the result is partially dependent on the cause. Causal reasoning is the process of drawing conclusions about a causal relationship based on the conditions under which a certain result occurs.

[0035] To address the issues presented in the previous technology, the present invention constructs a context-aware human behavior simulator by spontaneously learning the social characteristics of relationships between people. The simulator's training is unsupervised, significantly reducing the reliance on labeled data and, consequently, the amount of labeled data required. Furthermore, by training the relationship network based on the human behavior simulator, the system achieves higher accuracy, more efficient group detection, and is applicable to large scenarios.

[0036] Example 1

[0037] A self-supervised group detection method for large scenes with multiple objects, see Figure 1 , the method comprises the following steps:

[0038] Step 101: Modeling N persons in each t frames of a crowded multi-person scene video;

[0039] Step 102: Extracting characteristic information such as activity and time trajectory for each person;

[0040] Step 103: training an environment-aware human behavior simulator in a self-supervised manner, i.e., spontaneously learning natural human behavior in a group;

[0041] Step 104: Based on the idea of ​​causal reasoning, a simulator is used to train a relationship network to discover pairwise interpersonal relationships between multiple people;

[0042] Step 105: The parameters of the relational network are determined, and a simple fully connected (FC) layer-based network is added to implement the artificial group detection task.

[0043] In summary, the embodiment of the present invention implements self-supervised group detection of multiple objects in a large scene through the above steps 101-105.

[0044] Example 2

[0045] The solution in Example 1 is further introduced below with reference to specific calculation formulas and examples.

[0046] 1. Modeling N people in every t frames of multi-person scene videos

[0047] Given a video clip recording a crowded multi-person scene in T video frames, denoted as N here. For each t frame, all people in the scene are modeled as a directed graph G t = <V t ,R t >, where V t Indicates all A collection of represents the potential interpersonal relationship between role j and role i in frame t.

[0048] 2. Extract characteristic information such as activity and time trajectory for each person

[0049] First, extract the feature encoding vector for the tth person in the tth frame Where C is the number of feature channels. Encoded vector It can be extracted from t-frame segments and represented by human activities, time trajectories, etc.

[0050] We train a context-aware human behavior simulator in a self-supervised manner, i.e., spontaneously learning natural human behaviors in a swarm.

[0051] The goal of this embodiment of the present invention is to create a human behavior "simulator" that simulates reasonable human behavior within a crowd, within the constraints of social relationships, rules, and etiquette. For example, this includes the behavior and posture of people around them, as well as the spatial distribution of the crowd. This embodiment of the present invention is based on the assumption that everyone behaves logically and does not engage in strange behaviors, such as talking to someone, stepping on someone, or completely ignoring a greeting. This creates a common state of reasonable behavior.

[0052] The embodiment of the present invention assumes that the representation of a subject and its surrounding subjects can be modeled as an implicit distribution. Specifically, given a t-frame video clip of a character i and its neighbors, we can obtain:

[0053]

[0054] in, Represents the c main feature, Represents the characteristics of n neighbors of c entity. Definition is a feature cluster combining the central subject and the surrounding subjects, D represents an implicit distribution to be learned, n represents the set of n nearby neighbors, Nc represents the set of all objects in the entire scene, and T represents the total number of frames in the video.

[0055] In order to understand this distribution, the present invention proposes a self-supervision method. The basic idea is that if the state of the central subject (e.g., position, action) changes, the states of the surrounding subjects can coordinate and guide the central subject to recover. In this way, the present invention destroys the human and change it by space-time transformation to obtain Will be input into the restorer network and restored to The recovery network is represented as RecNet, and its training process can be expressed as:

[0056]

[0057] in, It is the central subject feature after time-space transformation; It is a feature cluster formed by combining the central subject and the surrounding subjects after time-space transformation.

[0058] Based on causal reasoning, a simulator is used to train a relational network to discover pairwise relationships between multiple people. Based on an environmentally aware behavioral simulator, the goal of this embodiment of the present invention is to spontaneously extract interpersonal relationships between a group of people. Inspired by causal reasoning, the basic concept of this embodiment of the present invention is to exclude surrounding agents j from a target agent i and observe the corresponding effects. This result can promptly reflect the relationship between agents i and j.

[0059] Given a feature with encoding at time t For subject i, assume that its neighbor set N i Contains N-1 subjects. The relationship between subject i and all subjects (including itself) is represented as a probability vector in For the target person and its neighbor j, the relationship probability is calculated through several steps As shown below:

[0060] First, delete the feature embedding of subject j, i.e., v j and Figure G t The edges involving subject j in . The current feature cluster (relative to i) is represented as As defined in formula (2), The feature embedding representing subject j is eliminated from the surrounding subjects.

[0061] The proposed behavior simulator is then used to recover the tampering signature cluster, i.e., From this, we can get the recovered features of subject i, namely

[0062] Finally, the relationship probability vector of j's influence on i is calculated, that is,

[0063] Repeat the above steps until all neighbors of i are calculated. Then, use the Softmax function to Normalized to meet And get the relationship vector of i

[0064] As mentioned above, the embodiment of the present invention has obtained the predicted relationship vector in a self-generated manner. In the following, the goal is to learn the human relationship extraction network, referred to as RelNet, which directly extracts the relationship vector from the feature group. The probability of learning the relationship is:

[0065]

[0066] in, Represents the predicted relation vector from RelNet. Finally, the parameters of the relation network are determined, and a simple fully connected (FC) layer-based network is added to implement the artificial group detection task.

[0067] In summary, the embodiment of the present invention first establishes a human behavior simulator module with environmental awareness to simulate the social behavior of multiple people in a crowd scene, and further establishes a relationship extractor module based on environmental elimination to model all two-way interpersonal relationships in the crowd. On the basis of the above two modules, an aggregated representation of interpersonal relationships for group detection tasks is established in a self-supervised manner. The present invention proposes a self-supervised method that can spontaneously learn the natural behavior of humans in a group without the need for a large amount of labeling; in addition, based on the idea of ​​causal reasoning, all two-way interpersonal relationships in the crowd are modeled. This method has the advantages of high group detection accuracy, no need for excessive labeling information, strong environmental adaptability, and low algorithm complexity.

[0068] Example 3

[0069] A self-supervised group detection device for large scenes with multiple objects, see Figure 2 The device includes: a processor 1 and a memory 2, wherein the memory 2 stores program instructions, and the processor 1 calls the program instructions stored in the memory 2 to enable the device to perform the following method steps:

[0070] Model N people in every t frames of a crowded multi-person scene video; extract feature information such as activity and time trajectory for each person;

[0071] Train an environment-aware human behavior simulator in a self-supervised manner, i.e., spontaneously learn natural human behaviors in a group;

[0072] Based on the idea of ​​causal reasoning, the relationship network is trained using an environmental perception human behavior simulator to discover pairwise interpersonal relationships between multiple people. The parameters of the relationship network are determined, and a network based on a fully connected layer is added to realize the artificial group detection task.

[0073] Among them, a self-supervised training of an environment-aware human behavior simulator, that is, spontaneously learning natural human behavior in a group, is as follows:

[0074] Destroyer The state of is changed by space-time conversion to obtain Will be input into the restorer network and restored to The recovery network is represented as RecNet, and the training process is expressed as:

[0075]

[0076] in, It is the central subject feature after time-space transformation; It is a characteristic cluster formed by combining the central subject and the surrounding subjects after time-space transformation;

[0077] in, Represents the c main feature, Represents the characteristics of the n neighbors of the c subject, and defines is a feature cluster combining the central subject and the surrounding subjects, D represents an implicit distribution to be learned, n represents the set of n nearby neighbors, Nc represents the set of all objects in the entire scene, and T represents the total number of frames in the video.

[0078] Among them, based on the idea of ​​causal reasoning, the relationship network is trained using the environmental perception human behavior simulator to discover the interpersonal relationships between multiple people. The parameters of the relationship network are determined as follows:

[0079] Given a coded feature at time t The subject i, the neighbor set N i Contains N-1 subjects, and the relationship between subject i and all subjects is represented as a probability vector in For the target person and its neighbor j, the relationship probability is calculated through several steps As shown below:

[0080] Delete the feature embedding of subject j, i.e. v j and Figure G t The edge involving subject j in the current feature cluster is represented as middle The feature embedding representing subject j is eliminated from the surrounding subjects;

[0081] The proposed behavior simulator is used to recover the tampering signature cluster, i.e., From this we get the recovered features of subject i Calculate the relationship probability vector of j's influence on i until all neighbors of i are calculated, and use the Softmax function to convert Normalized to meet And get the relationship vector of i

[0082] The goal is to learn a human relationship extraction network from feature groups The probability of learning the relationship is:

[0083]

[0084] in, Represents the predicted relation vector from RelNet, and finally determines the parameters of the relation network.

[0085] It should be noted here that the device description in the above embodiment corresponds to the method description in the embodiment, and the embodiment of the present invention will not be described in detail here.

[0086] The execution subjects of the above-mentioned processor 1 and memory 2 can be computers, single-chip microcomputers, microcontrollers and other devices with computing functions. In specific implementation, the embodiment of the present invention does not limit the execution subject and it is selected according to the needs of actual application.

[0087] Data signals are transmitted between the memory 2 and the processor 1 via the bus 3 , which will not be described in detail in the embodiment of the present invention.

[0088] References

[0089] [1]Jianchao Wu, Limin Wang, Li Wang, Jie Guo, and Gangshan Wu. Learningactor relation graphs for group activity recognition. InProceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition, pages 9964-9974, 2019.

[0090] [2]Mohamed A, Qian K, Elhoseiny M, et al.Social-STGCNN:ASocial Spatio-Temporal Graph Convolutional Neural Network for Human Trajectory Prediction[C] / / 2020IEEE / CVF[3]Conference on Computer Vision and Pattern Recognition(CVPR).IEEE, 2020.

[0091] Yao L,Chu Z,Li S,et al.A Survey on Causal Inference[J].2020.

Claims

1. A self-supervised group detection method for large-scale multi-object detection, characterized by: The method comprises: Model N people in every t frames of a crowded multi-person scene video; extract activity and time trajectory for each person; Train an environment-aware human behavior simulator in a self-supervised manner, i.e., spontaneously learn natural human behaviors in a group; Based on causal reasoning, we use an environmentally aware human behavior simulator to train a relational network to discover pairwise relationships between multiple people. We determine the parameters of the relational network and add a fully connected layer-based network to implement the artificial group detection task. The relationship network is trained based on the idea of ​​causal reasoning and using an environmental perception human behavior simulator to discover pairwise interpersonal relationships between multiple people. The parameters of the relationship network are specifically determined as follows: Given a coded feature at time t The subject i, the neighbor set N i Contains N-1 subjects, and the relationship between subject i and all subjects is represented as a probability vector in For the target person and its neighbor j, the relationship probability is calculated through several steps As shown below: Delete the feature embedding of subject j, i.e. v j and Figure G t The edge involving subject j in the current feature cluster is represented as The feature embedding representing subject j is eliminated from the surrounding subjects; The proposed behavior simulator is used to recover the tampering signature cluster, i.e., From this we get the recovered features of subject i Calculate the relationship probability vector of j's influence on i until all neighbors of i are calculated, and use the Softmax function to convert Normalized to meet And get the relationship vector of i The goal is to learn a human relationship extraction network from feature groups The probability of learning the relationship is: in, Represents the predicted relation vector from RelNet, and finally determines the parameters of the relation network.

2. The self-supervised group detection method for large-scale multi-object detection according to claim 1 is characterized in that: Train a context-aware human behavior simulator in a self-supervised manner, i.e., spontaneously learn natural human behavior in a group. Specifically: Destroyer The state of is changed by space-time conversion to obtain Will be input into the restorer network and restored to The recovery network is represented as RecNet, and the training process is expressed as: in, It is the central subject feature after time-space transformation; It is a characteristic cluster formed by combining the central subject and the surrounding subjects after time-space transformation; in, Represents the main characteristics of c, Represents the characteristics of the n neighbors of the c subject, and defines is a feature cluster combining the central subject and the surrounding subjects, D represents an implicit distribution to be learned, n represents the set of n nearby neighbors, Nc represents the set of all objects in the entire scene, and T represents the total number of frames in the video.

3. A self-supervised group detection device for large-scale multi-object detection, characterized in that: The device includes: a processor and a memory, wherein program instructions are stored in the memory, and the processor calls the program instructions stored in the memory to enable the device to execute the method steps according to any one of claims 1-2.

Images