Group behavior identification method based on asynchronous and synchronous relational graph

Abstract

The invention discloses a group behavior recognition method based on asynchronous and synchronous relational graphs, which comprises the following steps of: firstly, constructing an asynchronous relational graph model, capturing a time delay relationship in group behaviors by utilizing asynchronous characteristics among individuals, and constructing an asynchronous time delay relationship set; then, the features of the delay participants are shifted to synchronize with the features of the real-time participants, and the motion relationship and the position relationship are integrated into the relationship graph. Meanwhile, a synchronous relation graph model is constructed, the capturing capacity of interaction and motion differences among individuals is further optimized by introducing synchronous space interaction relation characteristics and local residual relation characteristics, and interference of non-key behaviors on group behavior recognition is restrained by combining local limiting conditions. And finally, constructing a graph aggregation module, introducing message passing with a multi-head attention mechanism, and realizing coupling of an asynchronous relation graph and a synchronous relation graph, so that the model can maintain relatively high group behavior recognition accuracy under the condition that the model contains asynchronous behavior individuals and non-key behavior individuals.

Description

Technical Field

[0001] The present application belongs to the technical field of group behavior recognition based on sensor data, and specifically relates to a group behavior recognition method based on asynchronous and synchronous relationship graphs. Background Art

[0002] Group action recognition (GAR) has broad application potential in many fields, including smart home, social behavior understanding, and sports activity analysis. Compared with the recognition of a single movement (such as running or jumping), group action recognition faces more complex challenges. For example, the difficulty of tasks such as group conversation or queuing lies not only in accurately identifying individual behaviors, but also in exploring the interactive relationship between different individuals.

[0003] At present, group behavior recognition based on different types of sensors has been explored, but most of the research still relies on cameras and computer vision technology. However, the use of cameras has obvious limitations due to privacy reasons and geographical restrictions. With the improvement of sensor manufacturing level, inertial sensors have obvious advantages over video data collection due to their small and convenient structure, which can continuously collect data and avoid being affected by obstacles and infringe less privacy. Modern smartphones and various wearable devices have built-in inertial sensor components, making it possible to wear them for a long time and continuously monitor behavioral characteristics. Therefore, group behavior recognition technology based on inertial sensor data has gradually developed and will have broader application prospects.

[0004] However, current research on group behavior recognition of sensors mainly focuses on group behavior recognition of synchronous behaviors. Existing methods mainly focus on processing behavioral relationships with synchronous time characteristics, while ignoring the problem of processing behavioral relationships with asynchronous time characteristics. In reality, group behaviors are not always synchronous, and asynchronous time characteristics (such as time delays) may affect the accurate learning of the interactive relationship between individuals, causing the current method to have a significant drop in accuracy when encountering asynchronous behavior.

[0005] Group behavior recognition of sensor data with asynchronous behavior faces the following challenges: first, how to capture the basic synchronous relationship of group behavior and effectively utilize the asynchronous relationship between individuals to capture the interactive relationship and asynchronous time delay in group behavior; second, how to fuse individual characteristics with the asynchronous characteristics of related individuals to enhance the expressiveness of individual characteristics and suppress the interference of asynchronous behavior; finally, it is necessary to explore how to effectively couple asynchronous relationships with synchronous relationships to enhance the expressiveness of the model, thereby achieving more accurate recognition and analysis of group behavior with asynchronous behavior. Summary of the invention

[0006] The purpose of this application is to provide a group behavior recognition method based on asynchronous and synchronous relationship graphs. This method obtains the time delay relationship in group behavior, synchronizes the characteristics of real-time and delayed participants, and integrates the movement and position relationship to improve the accuracy of group behavior recognition with asynchronous behavior. At the same time, the ability to capture interactions and movement differences between individuals is optimized, and the interference of non-critical behaviors is suppressed. Finally, by coupling the asynchronous and synchronous relationship graphs, the expressiveness of the model is enhanced, and the accuracy of group behavior recognition in asynchronous and interference situations is improved.

[0007] To achieve the above object, the technical solution adopted by the present invention is: A group behavior recognition method based on asynchronous and synchronous relationship graphs, comprising: The sensor data corresponding to each individual is divided into time series and input into the individual feature extraction module to extract individual behavior characteristics; Taking each individual in the group as a node, combined with the number of time series, the individual behavior characteristics are converted into a spatiotemporal graph and input into the asynchronous delay detection module to obtain the delay relationship between two individuals; In the feature fusion module, the individual behavior features are connected with the corresponding time delay shift features at the channel level to obtain the integrated features, and then the time delay relationship is fused with the integrated features to obtain the motion features of each individual; The motion features of each individual are input into the asynchronous relational reasoning module. In the relational reasoning module, the relation value is calculated through the motion relationship and position relationship between the two individuals, and then the asynchronous relation graph is obtained through the graph convolution network. The behavioral characteristics of each individual are input into the synchronous relationship reasoning module, the spatial interaction relationship characteristics and the residual relationship characteristics are calculated, the spatial interaction relationship characteristics are connected with the residual relationship characteristics to obtain the synchronous relationship graph node characteristics, and the graph convolution network is used to perform relationship reasoning to obtain the synchronous relationship graph; In the graph aggregation module, the synchronous relationship graph and the asynchronous relationship graph are aggregated to obtain the aggregated features; In the group behavior classification module, the aggregated features are input into the fully connected layer to obtain the group behavior recognition results.

[0008] Furthermore, the individual feature extraction module performs the following operations: The input sensor data is convolved and pooled through a convolutional neural network to obtain high-dimensional features; Input high-dimensional features into a bidirectional long short-term memory network to extract temporal features; Average pooling is performed on the temporal features to obtain significant features; Concatenate salient features with high-dimensional features to obtain local behavior features; The local behavior features are spliced ​​together to obtain the complete individual behavior features.

[0009] Furthermore, the asynchronous delay detection module performs the following operations: The features of nodes in the spatiotemporal graph are input into a linear feedforward classifier to obtain the energy for assigning individual behavior labels to nodes; The energy of assigning individual behavior labels to nodes is multiplied by the compatibility function and the spatiotemporal correlation function to obtain the pairwise energy of assigning the same label to two different nodes; Calculate the total energy of label assignment; The energy function is converted into a probability distribution of delay relations by minimizing the total energy of label assignments; Obtaining the time delay relationship with the maximum probability from the time delay relationship probability distribution to obtain the estimated time delay between the two individuals and the estimated time delay relationship probability; The time delay relationship probability is compared with the time delay relationship threshold to obtain the time delay relationship between the two individuals.

[0010] Furthermore, the time delay shift feature is obtained by performing a time shift operation on the individual behavior features of the delay participants according to the time delay obtained by the asynchronous time delay detection module.

[0011] Furthermore, the asynchronous relational reasoning module performs the following operations: The motion characteristics of the two individuals are linearly transformed respectively, and then normalized after transposition and multiplication to obtain the motion relationship between the two individuals; Calculate the positional relationship between the two individuals based on the position coordinates between the two individuals; The motion relationship between the two individuals and the position relationship between the two individuals are normalized to obtain a relationship value; Use graph convolutional networks to perform relational reasoning and obtain asynchronous relational graphs.

[0012] Furthermore, the synchronization relationship reasoning module performs the following operations: Connect the behavioral characteristics of an individual with those of other individuals, and then fuse them through a multi-layer perceptron to obtain the spatial interaction relationship characteristics centered on the individual; Calculate the difference between the behavioral characteristics of an individual and other individuals and the related interaction factors, then multiply the two and sum them to get the residual relationship characteristics; Connect the spatial interaction relationship features with the residual relationship features to obtain the synchronization relationship graph node features; The Euclidean distance between individuals is compared with the distance threshold to generate an adjacency matrix; Use graph convolutional networks to perform relational reasoning and obtain a synchronized relational graph.

[0013] Furthermore, the graph aggregation module performs the following operations: The node representation of the synchronous relationship graph is used as the edge of the asynchronous relationship graph, and the updated asynchronous features are obtained through the multi-head attention mechanism; The node representation of the asynchronous relationship graph is used as the edge of the synchronous relationship graph, and the updated synchronous features are obtained through the multi-head attention mechanism; The updated asynchronous features and the updated synchronous features, as well as the asynchronous features corresponding to the asynchronous relationship graph and the synchronous features corresponding to the synchronous relationship graph are connected and mapped to obtain aggregate features.

[0014] This application proposes a method for group behavior recognition based on asynchronous and synchronous relationship graphs. In the process of group behavior recognition, an asynchronous relationship graph model is first constructed, and the asynchronous features between individuals are used to capture the time delay relationship in group behavior and construct an asynchronous time delay relationship set. Then, the features of the delayed participants are shifted to synchronize with the features of the real-time participants, and the motion relationship and position relationship are integrated into the relationship graph. At the same time, a synchronous relationship graph model is constructed, and the ability to capture interactions and motion differences between individuals is further optimized by introducing synchronous spatial interaction relationship features and local residual relationship features, and local constraints are combined to suppress the interference of non-critical behaviors on group behavior recognition. Finally, a graph aggregation module is constructed, and message passing with a multi-head attention mechanism is introduced to achieve the coupling of asynchronous relationship graphs and synchronous relationship graphs, so that the model can maintain a high group behavior recognition accuracy even when it contains asynchronous behavior and non-critical behavior individuals. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] Figure 1 This is a flow chart of the group behavior recognition method based on asynchronous and synchronous relationship diagrams in this application.

[0016] Figure 2 This is a schematic diagram of the group behavior recognition network model structure of an embodiment of the present application. DETAILED DESCRIPTION

[0017] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0018] An embodiment of the present application, such as Figure 1 As shown, a group behavior recognition method based on asynchronous and synchronous relationship graphs is provided, including: Step S1: Divide the sensor data corresponding to each individual into time series and input them into the individual feature extraction module to extract individual behavior features.

[0019] This application constructs a group behavior recognition network model, such as Figure 2As shown, it includes an individual feature extraction module, an asynchronous delay detection module, a feature fusion module, an asynchronous relationship reasoning module, a synchronous relationship reasoning module, a graph aggregation module and a group behavior classification module.

[0020] For the same behavior, the movements at different parts of the body are different, so the motion data collected by sensors at different local positions of an individual are expressed in different forms. This application inputs the sensor data at different local positions of an individual into different networks for independent training to obtain more representative local features. Each individual can have sensors at multiple local positions, and there is no limit on the number of local positions. In addition, a body part can also have more sensors, and there is no limit on the dimension of the sensor data at each individual local position. In this application, the sensor data collected at a local position is 6-dimensional, and the sensors are located at two local positions: the wrist and the pocket of the lower limb body.

[0021] This application uses a sliding window to segment the sensor data corresponding to each individual. For example, a sliding window with a fixed length of 300 and an overlap rate of 50% is used. The sliding window is placed at the beginning of the time series, and then the sliding window moves backward to form another short time series with a length of 300. By analogy, a long time series sample is segmented into multiple short time series, and each short time series is divided into 6 time windows, each of which contains a subsequence of length 50. For a group of 5 people, the input of the model is the sensor data group of the 5 people in the same sliding window.

[0022] Because different participants in group activities may have asynchronous delays, dividing the sliding window helps to synchronize operations; experimental results show that the recognition effect is best when it is divided into six time windows. In addition, if the sliding window is set too small, it may not be possible to fully capture the entire process of the corresponding activity in one window.

[0023] In the experiment, the number of people in the group is set to a fixed 5, of which 1 to 4 have asynchronous behaviors. Through this setting, the recognition effect of the method of the present application when dealing with asynchronous behaviors in the group can be verified. In the experiment, two situations are considered: in situation 1, all 5 people belong to the same group; in situation 2, one or two individuals are randomly selected who do not belong to the group to simulate individual non-critical behaviors in the group, aiming to evaluate the ability of the method of the present application to suppress interference from non-critical behaviors.

[0024] This step describes the operations performed by the individual feature extraction module, including: Step 101: The input sensor data is convolved and pooled through a convolutional neural network to obtain high-dimensional features.

[0025] In this embodiment, the convolutional neural network performs convolution and pooling operations to obtain high-dimensional features. c s =[ c s 1 , c s 2 ,…, c s T ] , high-dimensional features still maintain temporal sequence. Indicates time window, output features at T time steps.

[0026] Step 102: Input the high-dimensional features into a bidirectional long short-term memory network to extract time series features.

[0027] In this embodiment, high-dimensional features Input into the BLSTM network (bidirectional long short-term memory network) to extract its timing features a s =[[ a s 1 ⃗ , a s 1 ⃖ ],[ a s 2 ⃗ , a s 2 ⃖ ],...,[ a s T ⃗ , a s T ⃖ ]] , represents the output features of the forward LSTM network at T time steps, is the output feature of the reverse LSTM network at T time steps.

[0028] Step 103: Perform average pooling on the temporal features to obtain significant features.

[0029] This example uses an average pooling layer to process the temporal features of the BLSTM network output. Extract the significant features and obtain the significant features o s =[avg( o s 1 ⃗ , o s 2 ⃗ ,..., o s T ⃗ ),avg( o s 1 ⃖ , o s 2 ⃖ ,..., o s T ⃖ )] .

[0030] Step 104: Concatenate the salient features with the high-dimensional features to obtain local behavior features.

[0031] In this embodiment, the significant features High-dimensional features extracted with convolutional neural networks (CNNs) Splicing to get local behavior features b tmp,s =[ c s , o s ] .

[0032] Step 105: splice the local behavior features to obtain complete individual behavior features.

[0033] In this embodiment, the sensor data at two local locations of the individual's wrist and lower limb body pocket are input into different CNN and BLSTM hybrid networks for independent training to obtain two local behavior features: b wri,s =[ c wri , o wri ] and b limb,s =[ c limb , o limb ] , and then the two local behavior features are spliced ​​to obtain the complete individual behavior feature b s =[ b wri,s , b limb,s ] .

[0034] Step S2: Taking each individual in the group as a node, combining the number of time series, converting the individual behavior characteristics into a spatiotemporal graph, inputting it into the asynchronous delay detection module, and obtaining the delay relationship between two individuals.

[0035] The asynchronous delay detection module is used to effectively utilize the asynchronous characteristics between individuals, capture the delay relationship in group behavior, and optimize and construct a set of asynchronous delay relationships through mean field reasoning to more accurately describe the time delay characteristics of individuals in the group. This optimization process can effectively reflect the delay dynamics in group behavior and provide more accurate group behavior data for further analysis.

[0036] This example creates a space-time graph To model the interaction of participants and simulate the evolution of spatiotemporal relationships between participants, , represents the node set of individuals in the group, where is the number of people in the group, Represents the number of time windows in the input data. For example, if there are 5 people in the group and a sliding window includes 6 time windows, then N is equal to 5 and M is equal to 6. is a set of edges connecting each pair of nodes across different time windows, and each node Represents the time window The individual below . Nodes of the space-time graph Features mainly represent the characteristics of individuals in a time window.

[0037] In the asynchronous delay detection module, the following operations are performed: Step 201: Input the features of the nodes in the spatiotemporal graph into a linear feedforward classifier to obtain the energy of assigning individual behavior labels to the nodes.

[0038] For nodes Features , by using the linear feedforward classifier Applied to node features To obtain the energy of assigning individual behavior labels to nodes ,Right now The linear feedforward classifier is a relatively mature technology in this field and will not be described in detail here.

[0039] Step 202: multiply the energy of assigning individual behavior labels to nodes by the compatibility function and the spatiotemporal correlation function to obtain the pairwise energy of assigning the same label to two different nodes.

[0040] In this embodiment, A compatibility function that takes the value 1 when , otherwise takes the value 0 and spatiotemporal correlation function , and After linear transformation Then pass through the linear feed-forward layer Multiply the result by , and get the same label assigned to the node and the pairwise costs of other nodes , as shown below: Among them, the compatibility function will be The value is 1 when it is true, otherwise it is 0.

[0041] Spatiotemporal correlation function as follows: in and It is a linear transformation with trainable weights. By learning the spatial relationship context and temporal dependency of each individual, the spatiotemporal correlation function can be obtained.

[0042] Step 203: Calculate the total energy of label allocation.

[0043] node The characteristic is expressed as , calculate the total energy of label allocation The formula is as follows: in represents the energy of assigning individual behavior labels to nodes, represents the pairwise energy of assigning the same label to two different nodes, Represents the time delay, that is, the number of time windows.

[0044] Step 204: Convert the energy function into a probability distribution of delay relations by minimizing the total energy of label allocation.

[0045] This step optimizes the inconsistency in label allocation by minimizing the total energy of label allocation. The energy function can be converted into a probability distribution of delay relationship by the following steps: , by using the mean field algorithm to update the probability distribution of the delay relationship to optimize: .

[0046] Step 205: Obtain the time delay relationship with the maximum probability from the time delay relationship probability distribution to obtain the estimated time delay between the two individuals and the estimated time delay relationship probability.

[0047] Specifically, according to the probability distribution of delay relationship Obtain the time delay relationship with the maximum probability to obtain the estimated time delay between two individuals and the estimated delay relationship probability : [△ s, P(y|B) ∗ ] = argm ax z P(y|B) .

[0048] Step 206: Compare the time delay relationship probability with the time delay relationship threshold to obtain the time delay relationship between the two individuals.

[0049] Specifically, compare the delay relationship probability Threshold value related to delay To detect the time delay relationship between two individuals ,as follows: .

[0050] Step S3: In the feature fusion module, the individual behavior features are connected with the corresponding time delay shift features at the channel level to obtain the integrated features, and then the time delay relationship is fused with the integrated features to obtain the motion features of each individual.

[0051] The delay relationship refers to the difference relationship between asynchronous behaviors in group behavior. In group behavior, delay participants refer to individuals whose behaviors or responses have time delays, while real-time participants are individuals without time delays. The activities of delay participants are not synchronized with those of other real-time participants, thus there is a delay relationship.

[0052] When there is a time delay relationship, the characteristics of the delayed participants need to be time-shifted to synchronize with the characteristics of the real-time participants. Individual behavioral characteristics of time-delay participants Perform time shift operation to obtain the time delay shift feature However, since the features after the input time window cannot be observed, the features in the last window need to be used to fill this part.

[0053] The time shift operation in this embodiment is: in, Represents the number of time windows in the input data.

[0054] Delay shift characteristics and individual behavioral characteristics of real-time participants Perform channel-level feature connections to obtain integrated features .

[0055] x ij,s =[ w i b i,s , w j b j,s s ℎ ift ] in, and Represents the weight parameter.

[0056] In group behavior, there may be multiple time delay relationships between real-time participants and different time delay participants. These different integration features have different effects on the overall real-time motion features, so the time delay relationship obtained by the asynchronous delay detection module is The corresponding integration features And trainable weights Multiply and add to get the fusion feature :

[0057] Finally, the motion characteristics of each participant are obtained , where participants are individuals in the group.

[0058] Step S4: Input the motion features of each individual into the asynchronous relational reasoning module. In the relational reasoning module, the relation value is calculated through the motion relationship and position relationship between the two individuals, and then the asynchronous relation graph is obtained through the graph convolution network.

[0059] The relational reasoning module of this embodiment characterizes the potential relationship of group behavior by integrating motion features and position coordinates, and models the motion relationship and position relationship respectively and integrates them into the relational graph to describe the interaction and spatial distribution characteristics between individuals.

[0060] The asynchronous relational reasoning module performs the following operations: Step 401: linearly transform the motion features of the two individuals respectively, transpose and multiply them, and then normalize them to obtain the motion relationship between the two individuals.

[0061] Specifically, the feature fusion module obtains individual Movement characteristics With the weight matrix Multiply and add to bias Adding the linear result , the feature fusion module is used to obtain individual Movement characteristics With the weight matrix Multiply and add to bias Adding the linear result ,Will After transposition transformation, Multiply and normalize to get the motion characteristics of two people The dot product similarity between them indicates the motion relationship between the participants. : in, express and Dimension.

[0062] Step 402: Calculate the positional relationship between the two individuals based on the position coordinates between the two individuals.

[0063] Specifically, given the position coordinates of two individuals , calculate the Euclidean distance between two individuals , the positional relationship between the two individuals is calculated by the following formula , integrating spatial structure information into the relationship graph: in Indicated by natural constant The exponential function with base , , It is the distance between the two farthest individuals within the range of group behavior, and is used to normalize the degree of influence of positional relationships to adapt to group behaviors of different ranges.

[0064] Step 403: normalize the motion relationship between the two individuals and the position relationship between the two individuals to obtain a relationship value.

[0065] In this embodiment, both motion characteristics and location information are considered to fully characterize the potential relationship of group behavior. and location relationship ,use The function is normalized to obtain the relationship value :

[0066] .

[0067] Step 404: Use the graph convolutional network to perform relational reasoning to obtain an asynchronous relational graph.

[0068] In this embodiment, the output of the graph convolutional network is the asynchronous feature corresponding to the asynchronous relationship graph. , as the output of the asynchronous relational reasoning module: in express Activation function, motion characteristics of all individuals , is the relationship value between individuals A collection of for The degree matrix of Normalized Matrix , is the weight matrix that needs to be learned.

[0069] Step S5: Input the behavioral characteristics of each individual into the synchronization relationship reasoning module, calculate the spatial interaction relationship characteristics and the residual relationship characteristics, connect the spatial interaction relationship characteristics with the residual relationship characteristics, obtain the synchronization relationship graph node characteristics, use the graph convolutional network to perform relationship reasoning, and obtain the synchronization relationship graph.

[0070] The synchronous relationship reasoning module further optimizes the ability to capture individual interactions and motion differences by introducing synchronous spatial interaction features and local residual relationship features. Combined with local constraints, it suppresses the interference of non-critical behaviors on group behavior recognition.

[0071] Synchronize the relational reasoning module and perform the following operations: Step 501: connect the behavioral characteristics of an individual with the behavioral characteristics of other individuals, and then fuse them through a multi-layer perceptron to obtain a spatial interaction relationship feature centered on the individual.

[0072] Specifically, the individual Behavioral characteristics and individuals Behavioral characteristics As input, the features and Features Perform connection operation [ , ] , using the function implemented by the multi-layer perceptron MLP Encode and sum the pairwise interaction features using the function implemented by MLP These encoded features are compared with the original features Fusion, to obtain individual The spatial interaction relationship characteristics of the center : r i o = f p b i + ∑ j =1 N f q ([ b i , b j ]) .

[0073] Step 502: Calculate the difference and related interaction factors between the behavior characteristics of the individual and other individuals, then multiply the two and sum them to obtain the residual relationship characteristics.

[0074] Specifically, calculate the individual Features and individuals Features Interaction factors : in, and are the learnable weight parameters and bias vectors, is the number of people in the group. d and m are subscripts and subscripts used to distinguish different parameters and have no special meaning.

[0075] Calculate individual Features Related participant characteristics The difference between Multiply and sum to get the residual relationship characteristics , indicating the difference in motion data between individual members: .

[0076] Step 503: Connect the spatial interaction relationship feature with the residual relationship feature to obtain the synchronization relationship graph node feature.

[0077] Specifically, the interactive relationship features and residual relationship characteristics Connect and get the updated synchronization graph node features : x i s =[ r i o , r i d ] .

[0078] Step 504: Compare the Euclidean distance between individuals with the distance threshold to generate an adjacency matrix.

[0079] In this embodiment, the Euclidean distance between individuals By comparing To limit the individuals within the range of interest, generate an adjacency matrix: in, Represents the Euclidean distance between two individuals. In group behavior modeling, sometimes local relationship information is more meaningful than global relationship.

[0080] Step 505: Use the graph convolutional network to perform relational reasoning to obtain a synchronization relation graph.

[0081] In this embodiment, a graph convolutional network is used to perform relational reasoning, and the output of the graph convolutional network synchronizes the features corresponding to the relational graph. , as the output of the synchronous relational reasoning module: in is the synchronization characteristics of all nodes before updating, for The degree matrix of Normalized Matrix , to prevent the network from experiencing gradient disappearance or gradient explosion, is the weight matrix that needs to be learned.

[0082] Step S6: In the graph aggregation module, the synchronous relationship graph and the asynchronous relationship graph are aggregated to obtain aggregate features.

[0083] The graph aggregation module realizes the coupling of asynchronous relationship graph and synchronous relationship graph by introducing message passing with multi-head attention mechanism. In the process of node update, the weighted feedback of attention head is used to optimize information transmission, so that asynchronous and synchronous relationship graphs can realize information mutual feedback in the spatial dimension. This method not only improves the expression ability of relationship graph model, but also enhances the accuracy of group behavior recognition.

[0084] In a specific embodiment, the graph aggregation module performs the following operations: Step 601: Use the node representation of the synchronous relationship graph as the edge of the asynchronous relationship graph, and obtain the updated asynchronous features through the multi-head attention mechanism.

[0085] The node representation of the synchronization relationship graph Used as the edge of the asynchronous relationship graph to represent the delay relationship Node (ie with asynchronous node The feature matrix of all connected edges As input, the updated asynchronous features are obtained through the multi-head attention mechanism : G a = 1 D as [ ℎ ea d 1 ( X ¯ m s , X ¯ m s , X ¯ m s ),..., ℎ ea d ℎ ( X ¯ m s , X ¯ m s , X ¯ m s )] W a in, represents the number of attention heads, is the number of edges connected to the asynchronous node, is the weight matrix.

[0086] Step 602: Use the node representations of the asynchronous relationship graph as the edges of the synchronous relationship graph, and obtain updated synchronization features through a multi-head attention mechanism.

[0087] Represent the nodes of the asynchronous relationship graph Used as the edge of the synchronization graph, the adjacency matrix Synchronous nodes (ie with the synchronization node The feature matrix of all connected edges As input, the updated synchronous features are obtained through the multi-head attention mechanism : G s = 1 D sa [ ℎ ea d 1 ( X ¯ n a , X ¯ n a , X ¯ n a ),..., ℎ ea d ℎ ( X ¯ n a , X ¯ n a , X ¯ n a )] W s in, represents the number of attention heads, is the number of edges connected to the sync node, is the weight matrix.

[0088] It should be noted that in the multi-head attention mechanism, each attention head is connected through the parameter matrix , , right , and Perform linear transformation. The specific process can be expressed as:

[0089] in is the index of the attention head. The parameter matrix of each head is not shared. yes The multi-head attention mechanism is a relatively mature technology in this field and will not be described here.

[0090] Step 603: Connect and map the updated asynchronous features and the updated synchronous features, as well as the asynchronous features corresponding to the asynchronous relationship graph and the synchronous features corresponding to the synchronous relationship graph to obtain aggregate features.

[0091] This step will update the asynchronous features and updated synchronization features , and the asynchronous characteristics corresponding to the asynchronous relationship diagram Synchronization characteristics corresponding to the synchronization relationship diagram Connect and map to generate the entire output aggregation feature of the graph aggregation module : G =[ G a , G s , X ¯ a , X ¯ s ] .

[0092] Step S7: In the group behavior classification module, the aggregated features are input into the fully connected layer to obtain the group behavior recognition result.

[0093] The group behavior classification module in this embodiment converts the output of the graph aggregation module Input the fully connected layer and perform Softmax activation to predict the final group behavior label.

[0094] The above-mentioned embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the invention patent. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the patent of the present application shall be subject to the attached claims.

Claims

1. A group behavior recognition method based on asynchronous and synchronous relationship graphs, characterized in that: The group behavior recognition method based on asynchronous and synchronous relationship graphs includes: The sensor data corresponding to each individual is divided into time series and input into the individual feature extraction module to extract individual behavior characteristics; Taking each individual in the group as a node, combined with the number of time series, the individual behavior characteristics are converted into a spatiotemporal graph and input into the asynchronous delay detection module to obtain the delay relationship between two individuals; In the feature fusion module, the individual behavior features are connected with the corresponding time delay shift features at the channel level to obtain the integrated features, and then the time delay relationship is fused with the integrated features to obtain the motion features of each individual; The motion features of each individual are input into the asynchronous relational reasoning module. In the relational reasoning module, the relation value is calculated through the motion relationship and position relationship between the two individuals, and then the asynchronous relation graph is obtained through the graph convolution network. The behavioral characteristics of each individual are input into the synchronous relationship reasoning module, the spatial interaction relationship characteristics and the residual relationship characteristics are calculated, the spatial interaction relationship characteristics are connected with the residual relationship characteristics to obtain the synchronous relationship graph node characteristics, and the graph convolution network is used to perform relationship reasoning to obtain the synchronous relationship graph; In the graph aggregation module, the synchronous relationship graph and the asynchronous relationship graph are aggregated to obtain the aggregated features; In the group behavior classification module, the aggregated features are input into the fully connected layer to obtain the group behavior recognition results.

2. The method for identifying group behavior based on asynchronous and synchronous relationship graphs according to claim 1, characterized in that: The individual feature extraction module performs the following operations: The input sensor data is convolved and pooled through a convolutional neural network to obtain high-dimensional features; Input high-dimensional features into a bidirectional long short-term memory network to extract temporal features; Average pooling is performed on the temporal features to obtain significant features; Concatenate salient features with high-dimensional features to obtain local behavior features; The local behavior features are spliced ​​together to obtain the complete individual behavior features.

3. The method for identifying group behavior based on asynchronous and synchronous relationship graphs according to claim 1, characterized in that: The asynchronous delay detection module performs the following operations: The features of nodes in the spatiotemporal graph are input into a linear feedforward classifier to obtain the energy for assigning individual behavior labels to nodes; The energy of assigning individual behavior labels to nodes is multiplied by the compatibility function and the spatiotemporal correlation function to obtain the pairwise energy of assigning the same label to two different nodes; Calculate the total energy of label assignment; The energy function is converted into a probability distribution of delay relations by minimizing the total energy of label assignments; Obtaining the time delay relationship with the maximum probability from the time delay relationship probability distribution to obtain the estimated time delay between the two individuals and the estimated time delay relationship probability; The time delay relationship probability is compared with the time delay relationship threshold to obtain the time delay relationship between the two individuals.

4. The method for identifying group behavior based on asynchronous and synchronous relationship graphs according to claim 4, characterized in that: The time delay shift feature is obtained by performing a time shift operation on the personal behavior characteristics of the delay participants through the time delay obtained by the asynchronous time delay detection module.

5. The method for identifying group behavior based on asynchronous and synchronous relationship graphs according to claim 1, characterized in that: The asynchronous relational reasoning module performs the following operations: The motion characteristics of the two individuals are linearly transformed respectively, and then normalized after transposition and multiplication to obtain the motion relationship between the two individuals; Calculate the positional relationship between the two individuals based on the position coordinates between the two individuals; The motion relationship between the two individuals and the position relationship between the two individuals are normalized to obtain a relationship value; Use graph convolutional networks to perform relational reasoning and obtain asynchronous relational graphs.

6. The method for identifying group behavior based on asynchronous and synchronous relationship graphs according to claim 1, characterized in that: The synchronization relationship reasoning module performs the following operations: Connect the behavioral characteristics of an individual with those of other individuals, and then fuse them through a multi-layer perceptron to obtain the spatial interaction relationship characteristics centered on the individual; Calculate the difference between the behavioral characteristics of an individual and other individuals and the related interaction factors, then multiply the two and sum them to get the residual relationship characteristics; Connect the spatial interaction relationship features with the residual relationship features to obtain the synchronization relationship graph node features; The Euclidean distance between individuals is compared with the distance threshold to generate an adjacency matrix; Use graph convolutional networks to perform relational reasoning and obtain a synchronized relational graph.

7. The method for identifying group behavior based on asynchronous and synchronous relationship graphs according to claim 1, characterized in that: The graph aggregation module performs the following operations: The node representation of the synchronous relationship graph is used as the edge of the asynchronous relationship graph, and the updated asynchronous features are obtained through the multi-head attention mechanism; The node representation of the asynchronous relationship graph is used as the edge of the synchronous relationship graph, and the updated synchronous features are obtained through the multi-head attention mechanism; The updated asynchronous features and the updated synchronous features, as well as the asynchronous features corresponding to the asynchronous relationship graph and the synchronous features corresponding to the synchronous relationship graph are connected and mapped to obtain aggregate features.

Images