Group behavior recognition method based on graph convolutional network and group relationship modeling

Abstract

This invention discloses a group behavior recognition method based on a graph convolutional network and group relationship modeling. Continuous sensor data collected by sensors at different local locations of individuals is segmented using a sliding window. For each individual, the local location sensor data is input into a convolutional neural network and a bidirectional long-short-term memory network to obtain individual behavior characteristics. Individual behavior correlations and individual position correlations are calculated, and an individual relationship graph is constructed. This graph is then input into a graph convolutional network to identify group behavior. This method fully exploits the characteristics of individual behaviors and the interactive relationships between individuals in the sensor data group, performing group feature-level characterization, thereby improving the accuracy and robustness of group behavior recognition.

Description

Technical Field

[0001] The present application belongs to the field of behavior recognition technology, and in particular relates to a group behavior recognition method based on graph convolutional networks and group relationship modeling. Background Art

[0002] Group behavior recognition has become a prominent research area. Group behavior refers to the overall behavior of a group of two or more individuals that influence and depend on each other. Group behavior recognition in complex scenarios is a challenging task because group behavior is not simply the sum of individual behaviors; instead, it requires inferring individual behaviors and the interactions between them. Therefore, it requires not only identifying individual behaviors within a group but also considering the complex interactions between individuals, conducting bottom-up behavioral analysis.

[0003] In recent years, the rapid development of the Internet of Things (IoT), the growth and proliferation of technologies such as wireless sensor networks and wearable devices, and the rapid development and application of various sensors have provided new data sources for group behavior recognition research. Currently, most group behavior research is conducted in the fields of vision and image processing. Compared to video and image processing, group behavior recognition based on sensor data is low-cost, unrestricted by geographical location, and less privacy-intrusive. Smart terminals now integrate a wide range of sensor devices, such as accelerometers, magnetometers, gyroscopes, and global positioning systems, making it feasible to apply these devices to group behavior recognition.

[0004] However, group behavior recognition based on sensor data is difficult to characterize group characteristics. Summary of the Invention

[0005] The purpose of this application is to provide a group behavior recognition method based on graph convolutional networks and group relationship modeling, which overcomes the problem of difficulty in characterizing group characteristics, can make full use of sensor data information, and improve the accuracy and robustness of group behavior recognition.

[0006] In order to achieve the above objectives, the technical solutions of this application are as follows:

[0007] A group behavior recognition method based on graph convolutional networks and group relationship modeling, including:

[0008] Receive continuous sensor data collected by sensors at different local positions of the individual, segment the continuous sensor data through a sliding window, and obtain individual local position sensor data after segmentation;

[0009] For each individual, the individual local position sensor data is input into a convolutional neural network to obtain a first feature, and then the first feature is input into a bidirectional long short-term memory network to obtain a second feature. The first feature and the second feature are concatenated to obtain a personal local behavior feature, and the individual local behavior features are concatenated to obtain an individual behavior feature.

[0010] Calculate individual behavior correlation based on individual behavior characteristics, calculate individual position correlation based on individual coordinates, and fuse individual behavior correlation and individual position correlation to obtain the interaction relationship between individuals;

[0011] The individual behavior characteristics and the interaction relationship between individuals are input into the graph convolutional network to extract the group interaction relationship characteristics. The group interaction relationship characteristics are then spliced with the individual behavior characteristics, and a maximum pooling operation is performed to obtain the group behavior characteristics used to describe the group behavior. The group behavior characteristics are then identified through a fully connected layer to obtain the group behavior recognition result.

[0012] Furthermore, the calculation of individual behavior correlation based on individual behavior characteristics, the calculation of individual position correlation based on individual coordinates, and the fusion of individual behavior correlation and individual position correlation to obtain the interaction relationship between individuals include:

[0013] For individuals i and j in the group, the individual behavior characteristics in the kth sliding window and Calculate the behavioral correlation between the two The formula is as follows:

[0014]

[0015] in express The covariance of Respectively The standard deviation of

[0016] Position correlation between individuals i and j within a population The calculation formula is as follows:

[0017]

[0018] in, is the average value of the Euclidean distance between individuals i and j in the kth sliding window;

[0019] The interaction relationship between individuals is obtained by fusing the individual behavior correlation and individual position correlation The fusion formula is expressed as follows:

[0020]

[0021] Among them, λ is the weight parameter.

[0022] Furthermore, the segmenting of the continuous sensor data by the sliding window further includes:

[0023] During segmentation, overlapping segmentation is performed according to a preset overlapping ratio.

[0024] Furthermore, the convolutional neural network includes a convolution layer and an average pooling layer.

[0025] Furthermore, the bidirectional long short-term memory network includes a forward long short-term memory network and a reverse long short-term memory network and their respective corresponding average pooling layers.

[0026] Furthermore, the individual behavior characteristics and the interaction relationship between individuals are input into a graph convolutional network, in which the individual behavior characteristics are used as nodes and the interaction relationship between individuals are used as edges between nodes.

[0027] This application proposes a group behavior recognition method based on graph convolutional networks and group relationship modeling, which fully mines the characteristics of individual behaviors and interaction relationships between individuals in the sensor data group, performs group feature-level representation, and improves the accuracy and robustness of group behavior recognition. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] Figure 1 This is a flowchart of the group behavior recognition method based on graph convolutional networks and group relationship modeling in this application.

[0029] Figure 2 This is a block diagram of group behavior recognition of the network model in the embodiment of the present application;

[0030] Figure 3 This is a flowchart of the process of extracting individual local behavior features in an embodiment of the present application.

[0031] Figure 4 Construct an individual interaction relationship block diagram for the embodiment of this application. DETAILED DESCRIPTION

[0032] In order to make the purpose, technical solutions and advantages of this application more clear, the following further describes this application in detail with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain this application and are not intended to limit this application.

[0033] Sensor data-based group behavior recognition uses sensors to acquire individual behavior data. This data is fed into the data of different individuals within a group, ultimately outputting the group behavior of the group. The sensors that collect this data primarily include accelerometers and gyroscopes. These sensors exhibit specific responses to different human movements. By using these two sensors to collect individual activity data from different parts of the body and combining it with the individual's coordinate location, we can fully leverage this sensor data to improve the accuracy and robustness of group behavior recognition.

[0034] In one embodiment, Figure 1 As shown in FIG, a group behavior recognition method based on graph convolutional network and group relationship modeling is provided, including:

[0035] Step S1: receiving continuous sensor data collected by sensors at different local positions of an individual, segmenting the continuous sensor data through a sliding window, and obtaining individual local position sensor data after segmentation.

[0036] Currently, various smart devices, such as smartphones and smartwatches, have become ubiquitous. These devices typically integrate various sensors, such as accelerometers, magnetometers, gyroscopes, and global positioning systems. This embodiment uses accelerometers and gyroscopes located on the wrist and in the pockets of the lower limbs as examples. This application processes the continuous sensor data collected by these sensors to identify group behavior.

[0037] This application segments continuous sensor data through a sliding window. In a specific example, the size of the sliding window is 200. The window size of 200 means that when the sensor frequency is 50 Hz, a total of 4 seconds of data is collected as one input data, that is, 50 data are sampled per second, and 200 data are sampled in four seconds as one input data.

[0038] Preferably, during segmentation, overlapping segmentation is performed according to a preset overlapping ratio.

[0039] For example, a window size of 200 is selected for 50% data overlap segmentation. 50% data overlap means repeated segmentation. When the window size of each input data is 200, the last 100 data are also used as the first 100 data of the next input data.

[0040] In a specific example, consider a wrist and lower limb pocket with smart devices, each equipped with an accelerometer and gyroscope. Each sensor provides x, y, and z-axis data, resulting in a total of six dimensions (i.e., six channels) for each body location. This continuous sensor data is segmented to obtain individual local location sensor data, which is subsequently input into a convolutional neural network. This data is represented as l = {l1, l2, ..., l6}, representing six dimensions.

[0041] In this embodiment, each individual has two pieces of individual local position sensor data. If sensors are also used to collect data on other parts of the individual's body, each individual can have more individual local position sensor data. This application does not limit the number of individual local position sensor data. In addition, a body part can also have more sensors. This application does not limit the dimensions of each individual local position sensor data, and it can have more dimensions.

[0042] This application identifies group behavior and needs to collect sensor data from all individuals in the group and input it into the network for group behavior identification. Figure 2 In the embodiment, it is assumed that there are 5 individuals in the group. The following embodiments are described using a group of 5 individuals as an example.

[0043] Step S2: For each individual, input the individual local position sensor data into a convolutional neural network to obtain a first feature, then input the first feature into a bidirectional long short-term memory network to obtain a second feature, concatenate the first feature and the second feature to obtain a personal local behavior feature, and concatenate the individual local behavior features to obtain an individual behavior feature.

[0044] In this step, the segmented individual local position sensor data is processed through a convolutional neural network (CNN) using one-dimensional convolutional pooling to obtain the first feature. The first feature is then passed through a bidirectional long short-term memory (BLSTM) neural network, using an average pooling layer to extract significant features from the state output of the BLSTM neural network to obtain the second feature.

[0045] like Figure 3As shown, the individual local position sensor data is first passed through a one-dimensional convolutional neural network, including convolutional layers and average pooling layers. In this embodiment, the convolution kernel size of the convolutional neural network (CNN) is 3, and the pooling layer window size is 2. This generates a high-dimensional feature with time series as the first feature. The first feature is then used as the input of a bidirectional long short-term memory (BLSTM) neural network. The BLSTM network includes a forward long short-term memory (LSTM) network and a reverse long short-term memory (LSTM) network, each with corresponding average pooling layers. After the BLSTM network fully learns the first feature, the network output state information is obtained. The average pooling layer then extracts features from the BLSTM output state information to obtain the second feature.

[0046] The first feature and the second feature are concatenated to obtain the individual local behavior feature F = [C, Y], where C represents the first feature and Y represents the second feature. Since the sensor data of the wrist and lower limb pocket positions (relative to the body parts) are trained independently in different networks, a more representative individual local behavior feature F can be obtained. arm =[C arm ,Y arm ] and F body =[C body ,Y body ], where F arm and F body Represent the individual wrist local features and body part local features, C arm and C body They represent the first features of the individual wrist and body parts after CNN pooling, Y arm and Y body They represent the second features corresponding to the individual wrist and body parts after BLSTM pooling.

[0047] In this embodiment, all individual local behavior features are spliced together to obtain individual behavior features. After the individual local position sensor data of the wrist (wrist sensor data) and the individual local position sensor data of the lower limb pocket position (body sensor data) are respectively subjected to the convolutional neural network and the bidirectional long short-term memory neural network, the individual local behavior features corresponding to the wrist (wrist features) and the individual local behavior features corresponding to the lower limb pocket position (body features) are obtained. Since the sensor data of the wrist and lower limb pocket positions are trained independently in different networks, more representative local features can be obtained. The two local features are then spliced together to obtain a complete individual behavior feature, and the individual behavior feature is used as the node input feature of the graph convolutional network. For example, the individual local behavior features are spliced together to obtain a complete individual behavior feature F per =[F arm ,F body ].

[0048] Step S3: Calculate individual behavior correlation based on individual behavior characteristics, calculate individual position correlation based on individual coordinates, and fuse individual behavior correlation and individual position correlation to obtain the interaction relationship between individuals.

[0049] The individual behavior characteristics are obtained through step S2, and the individual behavior correlation can be obtained by performing correlation analysis on the behavior characteristics between any two individuals in the group within a period of time.

[0050] For example, for individuals i and j in the group, the individual behavior characteristics in the kth sliding window are and The formula for calculating the behavioral correlation between the two is as follows:

[0051]

[0052]

[0053] in express The covariance of Respectively The standard deviation of .

[0054] The above calculations convert the behavioral characteristics of two individuals into behavioral correlation metrics. The larger the correlation metric value, the greater the possibility that the two individuals are participating in the same group behavior. On the contrary, the smaller the correlation metric value, the less likely it is that the two individuals are two individuals in a group performing the same behavior.

[0055] This step also uses individual coordinates to calculate the correlation of individual positions. First, the Euclidean distance between two individuals in the group is calculated. The calculation formula is as follows:

[0056]

[0057] Among them, d ij (t) represents the Euclidean distance between individuals i and j at time t, and They represent the coordinates of individual i at time t. When the distance between two individuals exceeds a certain level, it is considered that there is no interaction between them. A distance threshold of 5 is set, that is, when the distance between two individuals is greater than the threshold, the two individuals are considered not in the same group.

[0058] The average value of the Euclidean distance between individuals i and j in the kth sliding window is expressed as Since the degree of interaction between individuals is inversely proportional to the distance and decreases as the distance increases, the calculation formula for the position correlation between individuals i and j is as follows:

[0059]

[0060] Among them, e is a natural constant. Figure 4 As shown, for a group, the group includes individuals P1 to P5, and their individual behavior correlation and individual position correlation are represented by correlation graphs and distance graphs respectively. This application fuses individual behavior correlation and individual position correlation to obtain an individual interaction relationship graph to represent the interaction relationship between individuals.

[0061] The fusion formula is expressed as follows:

[0062]

[0063] Among them, λ is a weight parameter used to balance the importance of individual behavior relevance and position relationship. Indicates the importance of the characteristics of individual j to individual i.

[0064] For a group of 5 individuals, the interaction relationship between individuals is obtained by extracting the correlation between individual behaviors and the relationship between individual positions, and then fusing them through the interaction relationship fusion formula (5). Form a matrix G, such as Figure 4 The interaction relationships between individuals are shown in the relationship matrix. The interaction relationships between individuals are input into the graph convolutional network as edges between nodes.

[0065] Step S4: Input the individual behavior characteristics and the interaction relationship between individuals into the graph convolutional network to extract the group interaction relationship characteristics. Then, the group interaction relationship characteristics are spliced with the individual behavior characteristics, and a maximum pooling operation is performed to obtain the group behavior characteristics used to describe the group behavior. Then, a fully connected layer is used to identify the group behavior characteristics to obtain the group behavior recognition result.

[0066] This embodiment uses a graph convolutional network (GCN) to extract group interaction relationship features. Individual behavior features and individual interaction relationships form a relationship graph. Individual behavior features are used as nodes, and individual interaction relationships are used as edges between nodes. The graph convolutional network is input. A layer of graph convolutional network is used to perform a convolution operation on the neighborhood nodes of each node through the aggregation function of this layer to obtain the update result of the central node, and obtain n node features containing the potential semantic relationship between individuals. Then, an activation function is used for nonlinear transformation to obtain the group interaction relationship feature F. int .

[0067] The graph convolutional network can be expressed as follows:

[0068]

[0069] Among them, σ is the activation function, the activation function used is ReLU, F peris the individual behavior characteristic, G represents the relationship matrix (that is, the interaction relationship between individuals), and D is the degree matrix, which is calculated by G. The relationship matrix G can be normalized to prevent the network from gradient disappearance or gradient explosion. W1 is the graph convolution weight matrix, which needs to be learned through the network.

[0070] The group interaction relationship feature F obtained by a layer of graph convolutional network int , and n individual behavioral characteristics F per Splicing, performing maximum pooling to obtain the feature F used to describe group behavior gro , and use it as the input of the fully connected layer, and obtain the group behavior recognition result through the fully connected layer.

[0071] This application conducted experiments on group behavior recognition using a group behavior dataset constructed from the UT-Data dataset. The experiment used a fixed group size of 5 people, and tested both a fixed group size of 5 and a random group size of 2 to 5 people. Using 5-fold cross-validation, the accuracy, precision, recall, and F1 score were repeated 10 times to measure recognition performance. The experimental results are shown in Tables 1 and 2:

[0072]

[0073] Table 1

[0074]

[0075] Table 2

[0076] Table 1 and Table 2 show the test results for the cases of fixed and random group sizes, respectively. By comparing the two tables, it can be seen that the method of the present application is superior to the traditional group behavior recognition methods based on CNN network structure model, BLSTM network structure model, GCN network structure model, and CNN and BLSTM hybrid network structure model in classification, and has better robustness.

[0077] When the number of people in a group is random, the classification results of all methods are slightly worse than when the number of people in the test group is fixed, and the volatility of the results is also greater. The volatility of the proposed method is the smallest. The results show that the proposed method meets or exceeds existing group behavior recognition methods in all indicators, has better group behavior recognition results, and improves the accuracy and robustness of group behavior recognition.

[0078] The above-described embodiments merely represent several implementation methods of the present application. While the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the present invention. It should be noted that a person skilled in the art could make various modifications and improvements without departing from the spirit of the present application, all of which fall within the scope of protection of the present application. Therefore, the scope of protection of the present patent application shall be determined by the appended claims.

Claims

1. A group behavior recognition method based on graph convolutional networks and group relationship modeling, characterized by: The group behavior recognition method based on graph convolutional network and group relationship modeling includes: Receive continuous sensor data collected by sensors at different local positions of the individual, segment the continuous sensor data through a sliding window, and obtain individual local position sensor data after segmentation; For each individual, the individual local position sensor data is input into a convolutional neural network to obtain a first feature, and then the first feature is input into a bidirectional long short-term memory network to obtain a second feature. The first feature and the second feature are concatenated to obtain a personal local behavior feature, and the individual local behavior features are concatenated to obtain an individual behavior feature. Calculate individual behavior correlation based on individual behavior characteristics, calculate individual position correlation based on individual coordinates, and fuse individual behavior correlation and individual position correlation to obtain the interaction relationship between individuals; The individual behavior characteristics and the interaction relationship between individuals are input into the graph convolutional network to extract the group interaction relationship characteristics. The group interaction relationship characteristics are then spliced with the individual behavior characteristics and a maximum pooling operation is performed to obtain the group behavior characteristics used to describe the group behavior. The group behavior characteristics are then identified through a fully connected layer to obtain the group behavior recognition result. The calculation of individual behavior correlation based on individual behavior characteristics, the calculation of individual position correlation based on individual coordinates, and the fusion of individual behavior correlation and individual position correlation to obtain the interaction relationship between individuals include: For individuals i and j in the group, the individual behavior characteristics in the kth sliding window and Calculate the behavioral correlation between the two The formula is as follows: in express The covariance of Respectively The standard deviation of Position correlation between individuals i and j within a population The calculation formula is as follows: in, is the average value of the Euclidean distance between individuals i and j in the kth sliding window; The interaction relationship between individuals is obtained by fusing the individual behavior correlation and individual position correlation The fusion formula is expressed as follows: Among them, λ is the weight parameter.

2. The group behavior recognition method based on graph convolutional network and group relationship modeling according to claim 1 is characterized in that: The segmenting of the continuous sensor data by the sliding window further includes: During segmentation, overlapping segmentation is performed according to a preset overlapping ratio.

3. The group behavior recognition method based on graph convolutional network and group relationship modeling according to claim 1 is characterized in that: The convolutional neural network includes a convolution layer and an average pooling layer.

4. The group behavior recognition method based on graph convolutional network and group relationship modeling according to claim 1 is characterized in that: The bidirectional long short-term memory network includes a forward long short-term memory network and a reverse long short-term memory network and respective corresponding average pooling layers.

5. The method for group behavior recognition based on graph convolutional network and group relationship modeling according to claim 1, characterized in that: The individual behavior characteristics and the interaction relationship between individuals are input into the graph convolutional network, in which the individual behavior characteristics are used as nodes and the interaction relationship between individuals are used as edges between nodes.

Images