Full-scene group activity understanding method based on mixed hierarchical representation

Abstract

The invention provides a full-scene group activity understanding method based on mixed hierarchical representation, and mainly relates to a group activity understanding problem in the field of computer vision. The method comprises the steps of constructing a data set and a neural network framework, training a network, testing the network and the like. Wherein the neural network framework firstly extracts time sequence features, space environment features and the like to form rich individual features, and classifies individual actions; then, taking the individual features as nodes, reasoning information between the nodes at an individual interaction level and an action interaction level by using the proposed graph reasoning strategy, and fusing the information into group features; meanwhile, topological graph structures are constructed according to the position coordinates and the orientation respectively, a graph neural network is introduced for representation, and hierarchical features of a group structure are fused; and finally, splicing the group features and the group structure hierarchical features to identify the group activity type. According to the method, the multi-level representation of the group activity is fully realized, and the simultaneous identification of the individual actions of the group members and the whole activity type is realized.

Description

Technical Field

[0001] The present invention relates to the problem of video understanding in the field of computer vision, and in particular to a method for understanding group activities in a full scene based on mixed hierarchical representation. Background Art

[0002] Understanding group activities is one of the important tasks in the field of computer vision. This task aims to identify the actions of individual characters in the scene and the activities of the entire scene as a whole, and has great potential applications in many fields such as video surveillance, social robots, and urban management. Existing methods mainly mine general features of video data to identify individual actions and group activities, such as spatiotemporal information, individual interactions and other features, without considering the inherent multi-level feature representation of group activities in social scenarios. Starting from the multi-level representation of group activities, this patent constructs a multi-level understanding framework for group activities, comprehensively represents or infers the features of individual feature levels, individual interaction levels, action interaction levels, and group structure levels, and realizes accurate identification of group activities in social scenarios. Summary of the invention

[0003] The purpose of the present invention is to provide a method for understanding group activities in all scenarios based on hybrid hierarchical representation, comprehensively reasoning about the multi-level representation of social group activities, and realizing the recognition of the actions of group individuals and the activities of the group as a whole. For the convenience of explanation, first introduce the relevant professional terms and the concepts of the relevant expressions of this patent:

[0004] (1) Graph Neural Networks (GNNs): This network is designed to reason about graph structured data. The forward process usually includes two stages: message propagation and node update. This network includes variants such as graph convolutional network (GCN), gated graph neural network (GGNN), and graph attention network (GAT).

[0005] (2) Mixed hierarchical representation of group activities: This patent believes that group activities can be represented from four levels, namely individual feature hierarchical representation, individual interaction hierarchical representation, action interaction hierarchical representation and group structure hierarchical representation. Among them, individual feature hierarchical representation refers to the attributes, appearance, actions and other information of individuals in the group; individual interaction hierarchical representation refers to the relative relationship between individuals in the group in terms of body, position and gaze; action interaction hierarchical representation refers to the dependency relationship between individual action types in the group, such as the mutual dependency between speaking and listening to form the group activity of chatting; group structure hierarchical representation refers to the certain regularity and specific spatial pattern of group members in spatial position and face orientation due to different types of group activities.

[0006] The present invention specifically adopts the following technical solutions:

[0007] A hybrid hierarchical representation method for understanding group activities in all scenarios is proposed. The main features of this method are:

[0008] a. Identify group activities by comprehensively representing or reasoning about features at the individual feature level, individual interaction level, action interaction level, and group structure level;

[0009] b. Reasoning the features of individual interaction level and action interaction level through graph reasoning strategy with edge weights;

[0010] c. Characterize the group structure level by constructing a fully connected topological graph based on position and orientation;

[0011] The method mainly includes the following steps:

[0012] (1) Collect video data, annotate individual bounding boxes, individual action labels, group activity labels, location coordinates and orientation of the top-down perspective, and divide them into training and test sets;

[0013] (2) Construct a network framework for group activity understanding, including an individual feature extraction module, an interactive reasoning module, a group structure representation module, and a fusion classification module;

[0014] (3) The individual extraction module takes video frames, optical flow frames and individual bounding boxes as inputs, extracts temporal features of video frames and optical flow frames through temporal networks such as I3D, and extracts spatial environment features through semantic segmentation networks such as DeepLabV3+. Each feature is output in the form of a feature map. Then, each feature map is transformed to a uniform size and superimposed by interpolation. Then, the individual bounding box and ROI-Align modules are used to extract the individual features of each individual for individual action classification.

[0015] (4) Using individual features as nodes to construct a fully connected graph structure, the graph reasoning strategy with edge weights is used to infer the interaction information between individual features at the individual interaction level and the action interaction level. The individual features and the inferred features are concatenated as enhanced individual information, and multiple enhanced individual information are fused as group features.

[0016] (5) Using location coordinates and orientation as input, a graph neural network is used to represent group structure hierarchical features. Finally, group features and group structure hierarchical features are concatenated to identify the type of group activity.

[0017] (6) Design a multi-task joint loss function to optimize the network framework, including individual action loss, group activity loss, and edge weight loss;

[0018] (7) Use the training set and test set to iteratively train the network framework, and select the models that perform well in the test set for practical applications.

[0019] The beneficial effects of the present invention are: fully understanding the multi-level representation of group activities, while identifying the individual actions and overall activity types of group members. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] Figure 1 This is the overall schematic diagram of the network framework. DETAILED DESCRIPTION

[0021] The present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It is necessary to point out that the following embodiments are only used to further illustrate the present invention and cannot be understood as limiting the scope of protection of the present invention. Technical personnel familiar with the relevant field of technology may make some non-essential improvements and adjustments to the present invention based on the above invention content for specific implementation, which should still fall within the scope of protection of the present invention.

[0022] The full-scenario group activity understanding method based on hybrid hierarchical representation includes the following steps:

[0023] (1) Collect video data, annotate individual bounding boxes, individual action labels, group activity labels, location coordinates and orientation of the top-down perspective, and divide them into training and test sets;

[0024] (2) Construct a network framework for group activity understanding, including an individual feature extraction module, an interactive reasoning module, a group structure representation module, and a fusion classification module;

[0025] The individual extraction module takes video frames, optical flow frames and individual bounding boxes as inputs, extracts the temporal features of video frames (Input Video) and optical flow frames (Optic Flow) through temporal networks such as I3D, and extracts the spatial environment features of the center frame (Centre Frame) through semantic segmentation networks such as DeepLabV3+. Each feature is output in the form of a feature map; then, each feature map is transformed to a uniform size and superimposed by interpolation, and then the individual features (Individual Feature) of each individual are extracted using the individual bounding box and ROIAlign modules for individual action (IndividualAction) classification; this process can be expressed as follows:

[0026] v i =f roi ((M a ,M m ,M p )|b i )

[0027] Among them, M a 、M m 、M p They represent the temporal features of the video frame, the temporal features of the optical flow frame, and the spatial environment features of the center frame, respectively. i represents the bounding box coordinates of the i-th individual, f roi (·) represents the ROI Align module, v i represents the individual characteristics of the i-th individual;

[0028] The interactive reasoning module uses individual features as nodes to construct a fully connected graph structure, and uses a graph reasoning strategy with edge weights to infer the interactive information between individual features at the individual interaction level and the action interaction level. The features before and after reasoning are concatenated as enhanced individual information, and multiple enhanced individual information are fused as group features. Specifically, the weighted graph reasoning strategy first calculates the cosine similarity between nodes and normalizes it; i and v j For example, the calculation formula is:

[0029]

[0030] Then, the normalized cosine similarity is used as the weight on the edge to aggregate the neighboring node information with weights; taking node i in the lth round of node iteration as an example, let Θ be the learnable parameter, is the hidden state of the neighboring node in the current round, then the formula for node information aggregation is:

[0031]

[0032] Finally, the gated recurrent unit GRU is used to obtain the current hidden state of the node and aggregated neighbor node information Update the current node information for input. The calculation formula is:

[0033]

[0034] The group structure representation module takes the group structure representation (F-Formation) composed of position coordinates and orientation as input, uses a fully connected neural network to map the two-dimensional or three-dimensional position coordinates into a high-dimensional vector, and uses it as a node to build a fully connected graph structure; encodes the orientation as a one-hot encoding, and also uses a fully connected neural network to map it into a high-dimensional vector, and uses it as a node to build a fully connected graph structure; introduces a graph convolutional network for reasoning, and fuses the two graph structure nodes into two vectors in a point-by-point average manner;

[0035] The fusion classification module combines group features and group structure level features to identify the type of group activity;

[0036] (3) Design a multi-task joint loss function to optimize the network framework, including individual action loss, group activity loss and edge weight loss; the loss function is expressed as follows:

[0037]

[0038] Among them, e j,i and represents the predicted value and label value of the edge, and represents the predicted value and label value of group activities predicted using F-Formation information, and Represents the predicted value and label value of individual actions, O G and represents the final predicted group activity prediction value and label value, L edge (·),L ff (·),L ind (·),L gp (·) represent the corresponding loss functions respectively; in addition, the label value of the edge is determined by the action label between the individuals. If the action types of two individuals are the same, the label of the edge between them is 1, otherwise, it is 0;

[0039] (4) The network framework is iteratively trained using the training set and the test set, and the models that perform well in the test set are selected for practical applications.

Claims

1. A full-scenario group activity understanding method based on hybrid hierarchical representation, characterized by: a. Identify group activities by comprehensively representing or reasoning about features at the individual feature level, individual interaction level, action interaction level, and group structure level; b. Reasoning the features of individual interaction level and action interaction level through graph reasoning strategy with edge weights; c. Characterize the group structure level by constructing a fully connected topological graph based on position and orientation; The method mainly includes the following steps: (1) Collect video data, annotate individual bounding boxes, individual action labels, group activity labels, location coordinates and orientation of the top-down perspective, and divide them into training and test sets; (2) Construct a network framework for group activity understanding, including an individual feature extraction module, an interactive reasoning module, a group structure representation module, and a fusion classification module; (3) The individual extraction module takes video frames, optical flow frames and individual bounding boxes as inputs, extracts temporal features of video frames and optical flow frames through temporal networks such as I3D, and extracts spatial environment features through semantic segmentation networks such as DeepLabV3+. Each feature is output in the form of a feature map. Then, each feature map is transformed to a uniform size and superimposed by interpolation. Then, the individual bounding box and ROI-Align modules are used to extract the individual features of each individual for individual action classification. (4) Using individual features as nodes to construct a fully connected graph structure, a graph reasoning strategy with edge weights is used to infer the interaction information between individual features at the individual interaction level and the action interaction level. The features before and after reasoning are concatenated as enhanced individual information, and multiple enhanced individual information are fused as group features. (5) Using location coordinates and orientation as input, a graph neural network is used to represent group structure hierarchical features. Finally, group features and group structure hierarchical features are concatenated to identify the type of group activity. (6) Design a multi-task joint loss function to optimize the network framework, including individual action loss, group activity loss, and edge weight loss; (7) Use the training set and test set to iteratively train the network framework, and select the models that perform well in the test set for practical applications.

2. The method for understanding group activities in all scenarios based on hybrid hierarchical representation as claimed in claim 1, characterized in that: In step (4), the weighted graph reasoning strategy first calculates the cosine similarity between nodes and normalizes it; i and v j For example, the calculation formula is: Then, the normalized cosine similarity is used as the weight on the edge to aggregate the neighboring node information with weights; Taking node i in the lth round of node iteration as an example, let Θ be the learnable parameter, is the hidden state of the neighboring node in the current round, then the formula for node information aggregation is: Finally, the gated recurrent unit GRU is used to obtain the current hidden state of the node and aggregated neighbor node information Update the current node information for input. The calculation formula is:

3. The method for understanding group activities in all scenarios based on hybrid hierarchical representation as claimed in claim 1, characterized in that: In step (5), a fully connected neural network is used to map the two-dimensional or three-dimensional position coordinates into high-dimensional vectors, and a fully connected graph structure is constructed with them as nodes; the orientation is encoded as a one-hot encoding, and it is also mapped into a high-dimensional vector using a fully connected neural network, and a fully connected graph structure is constructed with them as nodes; a graph convolutional network is introduced for reasoning, and the two graph structure nodes are fused together in a point-by-point averaging manner to form two vectors, which are finally concatenated as features of the group structure level.

4. The method for understanding group activities in all scenarios based on hybrid hierarchical representation as claimed in claim 1, characterized in that: In step (6), the labels of the edges in the graph structure are assigned based on whether the action labels corresponding to the individual feature nodes are consistent; specifically, if two nodes have the same action label, the label of the edge between them is 1, otherwise it is 0, which is used to limit the information transfer between nodes with different action labels.

Images