Method for full-scene group activity understanding based on hybrid hierarchical representation

Abstract

The application provides a full-scene group activity understanding method based on a mixed hierarchical representation, and mainly relates to a group activity understanding problem in the field of computer vision. The method comprises the following steps: constructing a data set and a neural network framework, training a network, testing a network and the like. The neural network framework firstly extracts time sequence features, spatial environment features and the like to form rich individual features, and classifies individual actions; then, taking the individual features as nodes, information between the nodes in the individual interaction level and the action interaction level is inferred by using a proposed graph reasoning strategy, and is fused into group features; meanwhile, a topological graph structure is constructed by using position coordinates and orientations, and a graph neural network is introduced for representation, and is fused into group structure hierarchical features; finally, the group features and the group structure hierarchical features are spliced to identify a group activity type. The application fully represents a multi-level group activity, and simultaneously identifies individual actions of group members and a whole activity type.

Description

TECHNICAL FIELD

[0001] The present application relates to the video understanding problem in the field of computer vision, and in particular to a full-scene group activity understanding method based on mixed hierarchical representation. BACKGROUND

[0002] Group activity understanding is one of the important tasks in the field of computer vision. The task aims to identify the actions of individual characters and the activities of the whole scene, and has great potential applications in video surveillance, social robots, city management and many other fields. Existing methods mainly mine general features of video data to identify individual actions and group activities, such as spatiotemporal information, individual interaction and other features, without considering the inherent multi-level feature representation of group activities in social scenes. This patent starts from the multi-level representation of group activities, constructs a multi-level group activity understanding framework, comprehensively represents or reasons the individual feature level, individual interaction level, action interaction level and group structure level features, and realizes accurate identification of group activities in social scenes. SUMMARY

[0003] The purpose of the present application is to provide a full-scene group activity understanding method based on mixed hierarchical representation, which comprehensively reasons the multi-level representation of social group activities and realizes the identification of individual actions and group activities. For convenience of explanation, first introduce the related professional terms and the concept of related expressions in this patent:

[0004] (1) Graph Neural Networks (GNNs): This network aims to reason the neural network of graph structure data, and the forward process usually includes two stages of message propagation and node update. The network includes variants such as graph convolutional network (GCN), gated graph neural network (GGNN) and graph attention network (GAT).

[0005] (2) Group activity mixed level representation: The patent believes that group activity can be represented from four levels, which are individual feature level representation, individual interaction level representation, action interaction level representation and group structure level representation. Among them, the individual feature level representation refers to the information of the individual's own attributes, appearance and action in the group; the individual interaction level representation refers to the relative relationship between individuals in the group in terms of limbs, position and gaze; the action interaction level representation refers to the dependent relationship between the types of individual actions existing in the group, such as speech and listening to each other to constitute a chat group activity; the group structure level representation refers to the certain law and specific spatial pattern existing in the spatial position and face orientation of the group members due to the different types of group activities.

[0006] The application specifically adopts the following technical solutions:

[0007] A mixed level representation full-scene group activity understanding method is proposed, and the main features of the method are:

[0008] a. The group activity is identified by comprehensively representing or reasoning the individual feature level, individual interaction level, action interaction level and group structure level features;

[0009] b. The features of the individual interaction level and the action interaction level are reasoned by a graph reasoning strategy with edge weights;

[0010] c. The features of the group structure level are represented by constructing a full-connection topological graph based on position and orientation;

[0011] The method mainly includes the following steps:

[0012] (1) Collect video data, label individual bounding box, individual action label, group activity label, position coordinates and orientation of top-down perspective, and divide training set and test set;

[0013] (2) Construct a group activity understanding network framework, including an individual feature extraction module, an interaction reasoning module, a group structure representation module and a fusion classification module;

[0014] (3) The individual extraction module takes video frames, optical flow frames and individual bounding boxes as inputs, extracts the temporal features of video frames and optical flow frames through a temporal network such as I3D, extracts spatial environment features through a semantic segmentation network such as DeepLabV3+, and outputs each feature in the form of a feature map; then the features are transformed to a uniform size by interpolation and stacked together, and then the individual features of each individual are extracted by using the individual bounding box and the ROI-Align module for individual action classification;

[0015] (4) taking individual features as nodes to construct a full connection graph structure, using a graph reasoning strategy with edge weights to reason the interaction information between individual features in individual interaction levels and action interaction levels, splicing the individual features and the reasoned features as enhanced individual information, and fusing the multiple enhanced individual information as group features;

[0016] (5) taking position coordinates and orientations as inputs, using a graph neural network to represent group structure level features; finally, splicing the group features and the group structure level features to identify group activity types;

[0017] (6) designing a multi-task joint loss function for optimizing the network framework, including individual action loss, group activity loss and edge weight loss;

[0018] (7) using the training set and the test set to iteratively train the network framework, and selecting the model with good performance in the test set for practical application.

[0019] The beneficial effects of the present application are: fully understanding the multi-level representation of group activities, while identifying individual actions of group members and overall activity types. BRIEF DESCRIPTION OF DRAWINGS

[0020] Figure 1 It is the overall schematic diagram of the network framework. DETAILED DESCRIPTION

[0021] The following will be further described in detail in combination with the drawings and examples, and it is necessary to point out that the following examples are only used to further illustrate the present application and cannot be understood as limiting the protection scope of the present application. The skilled in the art can make some non-essential improvements and adjustments to the specific implementation of the present application according to the above inventive content, which should still belong to the protection scope of the present application.

[0022] The full-scene group activity understanding method based on hybrid level representation specifically includes the following steps:

[0023] (1) collect video data, label individual bounding boxes, individual action labels, group activity labels, position coordinates and orientations in the overhead perspective, and divide the training set and the test set;

[0024] (2) construct a group activity understanding network framework, including an individual feature extraction module, an interaction reasoning module, a group structure representation module and a fusion classification module;

[0025] The individual extraction module takes video frames, optical flow frames and individual bounding boxes as inputs, extracts the temporal features of the video frames (Input Video) and the optical flow frames (Optic Flow) through a time sequence network such as I3D, extracts the spatial environment features of the center frame (Centre Frame) through a semantic segmentation network such as DeepLabV3+, and each feature is output in the form of a feature map; then each feature map is transformed to a uniform size and stacked through interpolation, and then the individual features (Individual Feature) of each individual are extracted using the individual bounding box and the ROIAlign module for individual action (IndividualAction) classification; this process can be represented as follows:

[0026] v i =f roi ((M a ,M m ,M p )|b i )

[0027] wherein M a , M m , M p represent the temporal features of the video frames, the temporal features of the optical flow frames and the spatial environment features of the center frame respectively, b i represents the coordinates of the i-th individual's character bounding box, f roi (·) represents the ROI Align module, v i represents the individual features of the i-th individual.

[0028] The interaction reasoning module constructs a fully connected graph structure with individual features as nodes, uses a graph reasoning strategy with edge weights to reason the interaction information between individual features at the individual interaction level and the action interaction level, concatenates the features before and after reasoning as enhanced individual information, and fuses multiple enhanced individual information as group features; specifically, the weighted graph reasoning strategy first calculates the cosine similarity between nodes and normalizes it; taking nodes v i and v j as an example, the calculation formula is:

[0029]

[0030] Then take the normalized cosine similarity as the weight on the edge, and aggregate the neighbor node information with weight; taking node i in the l-th round of node iteration as an example, let Θ be the learnable parameter, be the hidden state of the neighbor node in the current round, then the formula for node information aggregation is:

[0031]

[0032] Finally, the gated recurrent unit (GRU) is used to update the current node information with the current hidden state of the node and the aggregated neighbor information The current node information is updated as input, and the calculation formula is:

[0033]

[0034] The group structure representation module takes the group structure representation (F-Formation) composed of position coordinates and orientation as input, maps the two-dimensional or three-dimensional position coordinates into high-dimensional vectors using a fully connected neural network, and constructs a fully connected graph structure with the high-dimensional vectors as nodes; the orientation is encoded into one-hot encoding, and a fully connected neural network is also used to map it into a high-dimensional vector, and a fully connected graph structure is constructed with the high-dimensional vector as nodes; a graph convolution network is introduced for inference, and the nodes of the two graph structures are fused into two vectors by point-by-point averaging;

[0035] The fusion classification module splices the group feature and the group structure hierarchical feature to identify the group activity type;

[0036] (3) A multi-task joint loss function is designed for optimizing the network framework, including individual action loss, group activity loss and edge weight loss; the loss function is represented as follows:

[0037]

[0038] wherein, e j,i and represent the predicted value and the label value of the edge, and represent the predicted value and the label value of the group activity predicted by the F-Formation information, and represent the predicted value and the label value of the individual action, O G and represent the predicted value and the label value of the final predicted group activity, L edge (·), L ff (·), L ind (·), L gp (·) represent the corresponding loss function respectively; in addition, the label value of the edge is determined by the action label between individuals, if the action types of two individuals are the same, the label of the edge between them is 1, otherwise, it is 0;

[0039] (4) The network framework is iteratively trained using the training set and the test set, and the model with good performance in the test set is selected for practical application.

Claims

1. A full-scene group activity understanding method based on mixed hierarchical representation, characterized in that: a. identifying group activities by comprehensive representation or reasoning of individual feature level, individual interaction level, action interaction level and group structure level features; wherein, individual feature level representation refers to appearance and action information of individuals in the group, individual interaction level representation refers to relative relationship between individuals in the group in terms of limbs, positions and gaze, action interaction level representation refers to dependency relationship between action types of individuals existing in the group, and group structure level representation refers to certain rules and specific spatial patterns in spatial positions and face orientations of group members due to different group activity types; b. reasoning individual interaction level and action interaction level features by graph reasoning strategy with edge weights; c. representing group structure level features by constructing a fully connected topological graph based on positions and orientations; the method mainly includes the following steps: (1) collecting video data, labeling individual bounding boxes, individual action labels, group activity labels, position coordinates and orientations in overhead perspective, and dividing training set and test set; (2) constructing a group activity understanding network framework, including individual feature extraction module, interaction reasoning module, group structure representation module and fusion classification module; (3) taking video frames, optical flow frames and individual bounding boxes as inputs, extracting time sequence features of video frames and optical flow frames respectively through time sequence network, extracting spatial environment features through semantic segmentation network, and outputting each feature in the form of feature map; then transforming each feature map to a uniform size by interpolation and stacking them together, and then extracting individual features of each individual by using individual bounding boxes and ROI-Align module for individual action classification; (4) constructing a fully connected graph structure with individual features as nodes, reasoning interaction information between individual features in individual interaction level and action interaction level by graph reasoning strategy with edge weights, concatenating features before and after reasoning as enhanced individual information, and fusing multiple enhanced individual information as group features; (5) taking position coordinates and orientations as inputs, representing group structure level features by graph neural network; finally, concatenating group features and group structure level features to identify group activity types; (6) designing a multi-task joint loss function for optimizing the network framework, including individual action loss, group activity loss and edge weight loss; (7) iteratively training the network framework using the training set and test set, and selecting the model with good performance in the test set for practical application.

2. The full-scene group activity understanding method based on mixed hierarchical representation according to claim 1, characterized in that:

3. The full-scene group activity understanding method based on mixed hierarchical representation according to claim 1, characterized in that: ​ ​ ​ ​ ​ ​ ​ ​ ​ ​ ​ ​ In step (4), the weighted graph reasoning strategy first calculates the cosine similarity between nodes and normalizes; take node v i and v j as an example, the calculation formula is: Then the normalized cosine similarity is taken as the weight of the edge, and the aggregation of the neighbor node information is carried out with weight. Taking the node i in the lth round of node iteration as an example, let Θ be a learnable parameter, is the hidden state of the neighbor node in the current round, and the formula for the aggregation of the node information is: Finally, the current hidden state of the node is updated by a gated recurrent unit (GRU) with the current hidden state of the node and the aggregated neighbor information and the aggregated neighbor information The current node information is updated by the input, and the calculation formula is as follows: ​ In step (5), a two-dimensional or three-dimensional position coordinate is mapped into a high-dimensional vector by using a fully connected neural network, and a fully connected graph structure is constructed with the high-dimensional vector as a node; the orientation is coded as a one-hot code, and is also mapped into a high-dimensional vector by using a fully connected neural network, and a fully connected graph structure is constructed with the high-dimensional vector as a node; a graph convolution network is introduced for inference, and two vectors are formed by fusing the nodes of the two graph structures in a point-by-point average manner respectively, and finally the two vectors are spliced to serve as a feature of a group structure level.

4. The full-scene group activity understanding method based on a hybrid level representation according to claim 1, wherein: In step (6), a label of an edge in the graph structure is assigned according to whether action labels corresponding to individual feature nodes are consistent; specifically, if two nodes have the same action label, a label of an edge between the two nodes is 1, otherwise, the label of the edge is 0, so as to limit information transmission between nodes with different action labels.

Images