A method for group behavior recognition based on sensor data based on joint attention mechanism

Abstract

The present invention discloses a sensor data group behavior recognition method based on a joint attention mechanism, constructing a network model including a data domain network and a semantic domain network, wherein the data domain network processes the local position sensor data of each individual in the group to obtain the data domain network group behavior characteristics, and then obtains the data domain network group behavior recognition result through a fully connected layer, and the semantic domain network processes the individual behavior label of each individual in the group to obtain the semantic domain network group behavior characteristics, and then obtains the semantic domain network group behavior recognition result through a fully connected layer. The loss function combines the data domain network and the semantic domain network, and performs joint training to obtain the network parameters of the data domain network. The present invention can effectively suppress the interference of non-critical behaviors in the group, and improve the accuracy and robustness of group behavior recognition.

Description

Technical Field

[0001] The present application belongs to the technical field of sensor data group behavior recognition, and in particular, relates to a sensor data group behavior recognition method with a joint attention mechanism. Background Art

[0002] Group behavior recognition has become a prominent research field. Group behavior refers to the overall behavior of a group of two or more mutually influential and interdependent individuals. Group behavior recognition in complex scenarios is a challenging task because group behavior is not a simple addition of individual behaviors, but requires inference of individual behaviors and the interactions between individuals. Therefore, it not only requires the recognition of individual behaviors in the group, but also the consideration of the complex interactions between individuals and bottom-up behavior analysis.

[0003] In recent years, the rapid development of the Internet of Things industry, the development and diffusion of technologies such as wireless sensor networks and wearable devices, along with the rapid development and application of various sensors, have provided new data sources for group behavior recognition research. At present, most group behavior research is conducted in the field of vision and image processing. Compared with videos and images, group behavior recognition based on sensor data is low-cost, not geographically restricted, and less privacy-intrusive. At present, a variety of sensor devices have been integrated on smart terminals, such as accelerometers, magnetometers, gyroscopes, global positioning systems, etc., which provides feasibility for the application of smart terminal devices in group behavior recognition.

[0004] However, group behavior recognition based on sensor data is difficult to characterize group characteristics, and when the input individual data does not belong to the same group, due to the lack of group detection, the input contains non-group individuals, that is, when non-critical behaviors are mixed in the group, the recognition effect will be reduced. Summary of the invention

[0005] The purpose of this application is to provide a sensor data group behavior recognition method based on a joint attention mechanism, which not only makes full use of sensor data information to overcome the difficulty of group feature characterization, but also makes full use of the semantic relationship between individual behavior labels and group behavior labels to improve the accuracy and robustness of group behavior recognition, and can effectively suppress the interference caused by non-critical individual behaviors in the group.

[0006] In order to achieve the above purpose, the technical solution of this application is as follows:

[0007] A sensor data group behavior recognition method based on a joint attention mechanism, comprising:

[0008] Step 1: construct a network model including a data domain network and a semantic domain network, wherein the data domain network is used to process the local position sensor data of each individual in the group to obtain the data domain network group behavior characteristics, and then obtain the data domain network group behavior recognition result through a fully connected layer, and the semantic domain network is used to process the individual behavior label of each individual in the group to obtain the semantic domain network group behavior characteristics, and then obtain the semantic domain network group behavior recognition result through a fully connected layer;

[0009] The data domain network is used to process the local position sensor data of each individual in the group to obtain the data domain network group behavior characteristics, including:

[0010] For each individual, the continuous sensor data collected by the sensor at the individual's local position is segmented through a sliding window to obtain the individual's local position sensor data after segmentation, the individual's different local position sensor data are input into a convolutional neural network to obtain a first feature, and then the first feature is input into a bidirectional long short-term memory network to obtain a second feature, the first feature and the second feature are spliced ​​to obtain a personal local behavior feature, all different individual local behavior features are spliced ​​to obtain an individual behavior feature, and the individual behavior feature is passed through a fully connected layer to obtain an individual behavior recognition result;

[0011] Calculate individual behavior correlation based on individual behavior characteristics, calculate individual position correlation based on individual coordinates, fuse individual behavior correlation and individual position correlation to obtain individual interaction relationship, input individual behavior characteristics and individual interaction relationship into the data domain graph convolution network, extract group interaction relationship characteristics, and concatenate group interaction relationship characteristics with individual behavior characteristics to obtain the third feature, which is then input into the attention module to learn and obtain data domain group behavior characteristics;

[0012] The semantic domain network is used to process the individual behavior label of each individual in the group to obtain the semantic domain network group behavior characteristics, including:

[0013] The individual behavior label is converted into a one-hot encoding vector, and the one-hot encoding vector and the individual position correlation are input into the semantic domain graph convolutional network to extract the semantic domain features. The semantic domain features are then input into the attention module to learn the semantic domain network group behavior features.

[0014] Step 2: Construct a loss function to combine the data domain network and the semantic domain network, and perform joint training to obtain the network parameters of the data domain network;

[0015] The loss function formula is as follows:

[0016]

[0017]

[0018]

[0019] Among them, L represents the joint loss, L p represents the classification loss of individual behavior recognition in the data domain network, L g represents the classification loss of group behavior recognition in the data domain network, L SPA represents the Euclidean distance between the data domain network and the semantic domain network, L KD represents the knowledge distillation loss, p i Represents individual behavior labels, represents the individual behavior recognition result of the data domain network, g i represents the group behavior label, P d represents the result of group behavior recognition in the data domain network, λ1 and λ2 are hyperparameters, represents the attention weight of the semantic domain network, represents the data domain network attention weight, P s Represents the results of semantic domain network group behavior recognition;

[0020] Step 3: Use the trained data domain network to process the local position sensor data of each individual in the group to be detected, and predict the group behavior recognition result corresponding to the group to be detected.

[0021] Furthermore, the calculation of individual behavior correlation based on individual behavior characteristics includes:

[0022] The covariance between individual behavioral characteristics is used as a measure of the correlation between individual behaviors.

[0023] Further, the calculating of individual position correlation based on individual coordinates includes:

[0024] Calculate the Euclidean distance between two objects. The formula is as follows:

[0025]

[0026] Among them, d ij (t) represents the Euclidean distance between individuals i and j at time t, and They represent the coordinates of individual i at time t respectively;

[0027] The average value of the Euclidean distance between individuals i and j in the kth sliding window is expressed as

[0028] The position correlation between individuals i and j in the kth sliding window is calculated by the following formula:

[0029]

[0030] Furthermore, the individual behavior correlation and the individual location correlation are integrated to obtain the individual interaction relationship, including:

[0031] Individual interaction value The fusion formula is as follows:

[0032]

[0033] Among them, λ is the weight parameter, Represents the correlation measure of individual behaviors.

[0034] Furthermore, the individual behavior characteristics and individual interaction relationships are input into a data domain graph convolutional network, wherein the individual behavior characteristics are used as nodes and the individual interaction relationships are used as edges between nodes.

[0035] Furthermore, the one-hot encoding vector and the individual position correlation are input into a semantic domain graph convolutional network to extract semantic domain features, wherein the one-hot encoding vector is used as a node and the individual position correlation is used as an edge between nodes.

[0036] This application proposes a method for identifying group behavior of sensor data based on a joint attention mechanism. The semantic domain network fully mines the potential semantic relationship between individual behavior and group behavior, and the data domain network fully mines the characteristics of individual behavior and interaction relationship between individuals in the sensor data group to perform group feature-level characterization. The semantic domain network is used to guide the allocation of attention weights of the data domain network, which can effectively suppress the interference of non-critical behaviors in the group and improve the accuracy and robustness of group behavior recognition. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] Figure 1 This is a flow chart of the sensor data group behavior recognition method based on the joint attention mechanism of this application;

[0038] Figure 2 This is a schematic diagram of the network model structure of an embodiment of the present application;

[0039] Figure 3 A schematic diagram of extracting individual local behavior characteristics using a data domain convolutional neural network and a bidirectional long short-term memory neural network in an embodiment of the present application;

[0040] Figure 4 This is a schematic diagram of splicing individual local features of the data domain network in the embodiment of the present application;

[0041] Figure 5 A schematic diagram of individual relationships in data domains is constructed for the embodiment of the present application. DETAILED DESCRIPTION

[0042] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0043] Sensor data group behavior recognition is to obtain individual behavior data through sensors, and use the data of different individuals in the group as input, and finally output the group behavior of the group. The sensors that collect data mainly include accelerometers and gyroscopes. When the human body moves in different states, the accelerometer and gyroscope will have specific manifestations. Through the above two sensors, individual activity data is collected at different parts of the body, and the coordinate position of the individual is added. The sensor data information can be fully utilized to improve the accuracy and robustness of group behavior recognition.

[0044] In one embodiment, Figure 1 As shown, a sensor data group behavior recognition method based on a joint attention mechanism is provided, including:

[0045] Step S1, constructing a network model including a data domain network and a semantic domain network, wherein the data domain network is used to process the local position sensor data of each individual in the group to obtain the data domain network group behavior characteristics, and then obtain the data domain network group behavior recognition result through a fully connected layer, and the semantic domain network is used to process the individual behavior label of each individual in the group to obtain the semantic domain network group behavior characteristics, and then obtain the semantic domain network group behavior recognition result through a fully connected layer.

[0046] The network model constructed in this embodiment is as follows Figure 2 As shown in Figure 1, it includes data domain network and semantic domain network. The semantic domain network is mainly used to influence the allocation of data domain network parameters when training the network model.

[0047] In a specific embodiment, the data domain network is used to process the local position sensor data of each individual in the group to obtain the data domain network group behavior characteristics, including:

[0048] For each individual, the continuous sensor data collected by the sensor at the individual's local position is segmented through a sliding window to obtain the individual's local position sensor data after segmentation, the individual's different local position sensor data are input into a convolutional neural network to obtain a first feature, and then the first feature is input into a bidirectional long short-term memory network to obtain a second feature, the first feature and the second feature are spliced ​​to obtain a personal local behavior feature, all different individual local behavior features are spliced ​​to obtain an individual behavior feature, and the individual behavior feature is passed through a fully connected layer to obtain an individual behavior recognition result;

[0049] The individual behavior correlation is calculated based on the individual behavior characteristics, and the individual position correlation is calculated based on the individual coordinates. The individual behavior correlation and the individual position correlation are fused to obtain the individual interaction relationship. The individual behavior characteristics and the individual interaction relationship are input into the data domain graph convolutional network to extract the group interaction relationship characteristics, and the group interaction relationship characteristics are spliced ​​with the individual behavior characteristics to obtain the third feature. The third feature is then input into the attention module to learn and obtain the data domain group behavior characteristics.

[0050] At present, various smart devices have become popular, such as smart phones, smart watches, etc. These smart devices usually integrate various sensors, such as accelerometers, magnetometers, gyroscopes, global positioning systems, etc. In this embodiment, accelerometers and gyroscope sensors at the wrist and lower limb pockets are used as examples for illustration. The present application processes the continuous sensor data collected by the sensor and identifies group behavior.

[0051] The sensors at different local positions of individuals collect continuous sensor data, which needs to be segmented through a sliding window to obtain the sensor data of individual local positions. In a specific example, the size of the sliding window is 200. The window size of 200 means that when the sensor frequency is 50Hz, a total of 4s of data is collected as an input data, that is, 50 data are sampled per second, and 200 data are sampled in four seconds as an input data. During segmentation, overlapping segmentation is also performed according to a preset overlapping ratio.

[0052] For example, select a window size of 200 for 50% data overlap segmentation. 50% data overlap means repeated segmentation. When the window size of each input data is 200, the last 100 data are also used as the first 100 data of the next input data.

[0053] After processing, the continuous sensor data collected by sensors at different local positions are used to obtain individual sensor data at different local positions. For example, if there are smart devices at the wrist and lower limb pockets, each smart device has an accelerometer and a gyroscope sensor, and each sensor has x, y, and z three-axis data, then each local position of the body has a total of 6-dimensional data (i.e., 6 channels). These continuous sensor data are segmented to obtain the individual local position sensor data that is subsequently input into the convolutional neural network, expressed as l = {l1, l2, ..., l6}, which is 6-dimensional data.

[0054] In this embodiment, each individual has two pieces of individual local position sensor data. If other parts of the individual's body also have sensors to collect data, each individual can have more individual local position sensor data, and this application does not limit the number of individual local position sensor data. In addition, a body part can also have more sensors, and this application does not limit the dimension of each individual local position sensor data, and it can have more dimensions.

[0055] In order to identify group behavior, this application needs to collect sensor data from all individuals in the group and input it into the data domain network. Figure 2 In the embodiment, it is assumed that there are 5 individuals in the group, and the group of 5 individuals is used as an example for explanation in the following embodiments.

[0056] For each individual, the individual local position sensor data is input into the convolutional neural network to obtain the first feature, and then the first feature is input into the bidirectional long short-term memory network to obtain the second feature. The first feature and the second feature are concatenated to obtain the personal local behavior feature, and all the individual local behavior features are concatenated to obtain the individual behavior feature.

[0057] In this step, the individual local position sensor data obtained after segmentation is processed by one-dimensional convolution pooling through the convolutional neural network CNN to obtain the first feature. The first feature is then passed through the bidirectional long short-term memory neural network BLSTM, and the average pooling layer is used to extract the significant features of the state output by the bidirectional long short-term memory neural network to obtain the second feature.

[0058] like Figure 3 As shown, the individual local position sensor data is first passed through a one-dimensional convolutional neural network, including a convolution layer and an average pooling layer. In this embodiment, the convolution kernel size of the convolutional neural network CNN is 3, and the pooling layer window size is 2, to obtain a high-dimensional feature with time series as the first feature. The first feature is used as the input of a bidirectional long short-term memory neural network BLSTM. The bidirectional long short-term memory network BLSTM includes a forward long short-term memory network and a reverse long short-term memory network and their corresponding average pooling layers. After the bidirectional long short-term memory neural network fully learns the first feature, the state information of the network output is obtained. The state information output by the bidirectional long short-term memory neural network is extracted through the average pooling layer to obtain the second feature.

[0059] The first feature and the second feature are concatenated to obtain the individual local behavior feature F = [C, Y], where C represents the first feature and Y represents the second feature. Since the sensor data of the wrist and lower limb pocket positions (relative to the body parts) are trained independently in different networks, a more representative individual local behavior feature F can be obtained. arm =[C arm ,Yarm ] and F body =[C body ,Y body ], where F arm and F body Represent the individual wrist local features and body local features, C arm and C body They represent the first features of the wrist and body parts of the individual after CNN pooling, Y arm and Y body They respectively represent the second features corresponding to the individual wrist and body parts after BLSTM pooling.

[0060] like Figure 4 As shown, in this embodiment, all individual local behavior features are spliced ​​together to obtain individual behavior features. After the individual local position sensor data of the wrist (wrist sensor data) and the individual local position sensor data of the lower limb pocket position (body sensor data) are respectively subjected to the convolutional neural network and the bidirectional long short-term memory neural network, the individual local behavior features corresponding to the wrist (wrist features) and the individual local behavior features corresponding to the lower limb pocket position (body features) are obtained. Since the sensor data of the wrist and lower limb pocket positions are independently trained in different networks, more representative local features can be obtained, and then the two local features are spliced ​​together to obtain a complete individual behavior feature. For example, the individual local behavior features are spliced ​​together to obtain a complete individual behavior feature F per =[F arm ,F body ].

[0061] It should be noted that after obtaining the individual behavior features, a fully connected layer (classifier) ​​can be used to obtain the individual behavior recognition results. Classifying the features through a fully connected layer to obtain the classification results is a relatively mature technology in this field and will not be described here.

[0062] In this implementation, individual behavior correlation is calculated based on individual behavior characteristics, individual position correlation is calculated based on individual coordinates, and individual behavior correlation and individual position correlation are fused to obtain individual interaction relationship.

[0063] After obtaining the individual behavior characteristics, the individual behavior correlation can be obtained by performing correlation analysis on the behavior characteristics between any two individuals in the group within a period of time. In this embodiment, the covariance between the individual behavior characteristics is used as the individual behavior correlation measurement value.

[0064] For example, for individuals i and j in the group, the individual behavior characteristics in the kth sliding window are and The formula for calculating the behavioral correlation between the two is as follows:

[0065]

[0066]

[0067] in express The covariance of Respectively The standard deviation of . Represents the correlation measure of individual behaviors.

[0068] The above calculations convert the behavioral characteristics of two individuals into behavioral correlation metrics. The larger the correlation metric value, the greater the possibility that the two individuals participate in the same group behavior. On the contrary, the smaller the correlation metric value, the less likely it is that the two individuals are two individuals in a group performing the same behavior.

[0069] This embodiment also calculates individual position correlation based on individual coordinates, and similarly divides individual coordinate data of continuous time by using a sliding window to calculate individual position correlation.

[0070] First, calculate the Euclidean distance between two individuals in the group. The calculation formula is as follows:

[0071]

[0072] Among them, d ij (t) represents the Euclidean distance between individuals i and j at time t, and They represent the coordinates of individual i at time t. When the distance between two individuals exceeds a certain level, it can be considered that there is no interaction between the two individuals. A distance threshold is set to 5, that is, when the distance between the individuals is greater than the threshold, the two individuals are considered not to be in the same group.

[0073] The average value of the Euclidean distance between individuals i and j in the kth sliding window is expressed as Since the degree of interaction between individuals is inversely proportional to the distance and decreases with the increase of distance, the individual position correlation A s The position correlation between individuals i and j in The calculation formula is as follows:

[0074]

[0075] Where, e is a natural constant. Individual position correlation A s is a matrix that includes the position correlations among all individuals.

[0076] like Figure 5As shown, for a group, the group includes individuals P1 to P5, and their individual behavior correlation and individual position correlation are represented by correlation graph and distance graph respectively. The present application fuses the individual behavior correlation and individual position correlation to obtain an individual interaction relationship graph to represent the individual interaction relationship.

[0077] Individual interaction value The fusion formula is expressed as follows:

[0078]

[0079] Among them, λ is a weight parameter, which is used to balance the importance of individual behavior relevance and position relationship. Indicates the importance of the characteristics of individual j to individual i.

[0080] For a group of 5 individuals, the individual behavior correlation and individual position relationship are extracted, and then the individual interaction relationship is obtained by fusion through the interaction relationship fusion formula. To form a matrix G, such as Figure 5 The relationship matrix is ​​shown.

[0081] like Figure 2 As shown in the figure, individual behavior characteristics and individual interaction relationships are input into the graph convolutional network to extract group interaction relationship characteristics. Then, the group interaction relationship characteristics are concatenated with the individual behavior characteristics, and attention-weighted fusion is performed to obtain group behavior characteristics used to describe group behavior. Then, a fully connected layer is used to identify the group behavior characteristics to obtain the group behavior recognition result.

[0082] This embodiment uses a graph convolutional network GCN to extract group interaction relationship features. Individual behavior features and individual interaction relationships form a relationship graph. Individual behavior features are used as nodes, and individual interaction relationships are used as edges between nodes. The graph convolutional network is input, and a layer of graph convolutional network is used. The aggregation function of this layer is used to perform a convolution operation on the neighborhood nodes of each node to obtain the update result of the central node, and n node features containing potential semantic relationships between individuals are obtained. Then, the ReLU activation function is used for nonlinear transformation to obtain N individual interaction relationship features F int,n .

[0083] The data domain graph convolutional network can be expressed as follows:

[0084]

[0085] Among them, σ is the activation function, the activation function used is ReLU, F per,n is the N individual behavior features, G represents the relationship matrix (that is, the individual interaction relationship), D is the degree matrix, which is the weight sum of the edges connecting the target node and each neighbor in G, calculated by G, and obtained by The relationship matrix G can be normalized to prevent the network from experiencing gradient disappearance or gradient explosion, W g It is the data domain graph convolution weight matrix, which needs to be learned through the network.

[0086] The group interaction feature F obtained by a layer of graph convolutional network int,n After that, with n individual behavior characteristics F per,n Splicing, get the third feature, input into the attention module to calculate the correlation of individual feature vectors And use softmax to normalize to get the data domain network attention weight Then perform weighted fusion of group behavior features v d Among them, the attention module is expressed by the following formula:

[0087]

[0088]

[0089]

[0090]

[0091]

[0092]

[0093] in and is the weight matrix to be learned, (K d ) T Denotes the matrix K d The transpose of .

[0094] In a specific embodiment, the semantic domain network is used to process the behavior label of each individual in the group to obtain the semantic domain network group behavior characteristics, including:

[0095] The individual behavior label is converted into a one-hot encoded vector, and the one-hot encoded vector and the individual position correlation are input into the semantic domain graph convolutional network to extract the semantic domain features. The semantic domain features are then input into the attention module to learn the semantic domain network group behavior features.

[0096] Specifically, the individual behavior labels in the group are encoded into one-hot vectors where f oh,n ∈R C, C is the number of individual behavior categories, and N is the number of individuals. Then the position correlation is calculated through the coordinates, input into the semantic domain graph convolutional network to extract features, and then input into the attention module to learn the semantic domain network group behavior features.

[0097] This step uses individual coordinates to calculate individual position correlation A s , first calculate the Euclidean distance between two individuals in the group, the calculation formula is as follows:

[0098]

[0099] Among them, d ij (t) represents the Euclidean distance between individuals i and j at time t, and They represent the coordinates of individual i at time t. When the distance between two individuals exceeds a certain level, it can be considered that there is no interaction between the two individuals. A distance threshold is set to 5, that is, when the distance between the individuals is greater than the threshold, the two individuals are considered not to be in the same group.

[0100] The average value of the Euclidean distance between individuals i and j in the kth sliding window is expressed as Since the degree of interaction between individuals is inversely proportional to the distance and decreases with the increase of distance, the individual position correlation A s The calculation formula of the position correlation between individuals i and j in is as follows:

[0101]

[0102] Among them, e is a natural constant.

[0103] The semantic domain graph convolution in this embodiment is expressed by the following formula:

[0104]

[0105] Among them, the degree matrix D is A s The sum of the weights of the edges connecting the target node and each neighbor, W s is the weight matrix of the semantic domain graph convolution. oh,n After graph convolution, we get O s The input is sent to the attention module, and a weight distribution of the attention coefficient is learned through self-attention mechanism training. The weight distribution is added to the original features, taking into account the influence of all individuals, and selectively focusing on the behavioral characteristics of key people that make important contributions to the final group behavior recognition.

[0106] The attention module of this embodiment is expressed by the following formula:

[0107]

[0108]

[0109]

[0110]

[0111]

[0112]

[0113] in and is the weight matrix to be learned, (K s ) T Denotes the matrix K s The transpose of is the correlation of individual behaviors, and is normalized using softmax to obtain the attention weight of the semantic domain network The group behavior feature v obtained by weighted fusion of attention s .

[0114] Step S2: construct a loss function to unite the data domain network and the semantic domain network, and perform joint training to obtain the network parameters of the data domain network.

[0115] When training the network model, this embodiment combines the two networks of the data domain and the semantic domain through the loss function, retains the attention learned in the semantic domain as much as possible, and guides the allocation of attention weights in the data domain. The loss function calculation is as follows:

[0116]

[0117]

[0118]

[0119] Among them, L represents the joint loss, L p represents the classification loss of individual behavior recognition in the data domain network, L g represents the classification loss of group behavior recognition in the data domain network, L SPA represents the Euclidean distance between the data domain network and the semantic domain network, L KD represents the knowledge distillation loss, p i Represents individual behavior labels, represents the individual behavior recognition result of the data domain network, g i represents the group behavior label, P d represents the result of group behavior recognition in the data domain network, λ1 and λ2 are hyperparameters, represents the attention weight of the semantic domain network, represents the data domain network attention weight, Ps Represents the results of semantic domain network group behavior recognition.

[0120] λ1 and λ2 are hyperparameters used to balance the influence of the two losses that the semantic domain can guide the data domain. SPA The aim is to make the attention of the data domain network closer to the attention of the semantic domain. The Euclidean distance between the attention of the semantic domain and the attention of the data domain is calculated by the mean square distance. The closer the values ​​of the two are, the smaller the calculated mean square distance will be.

[0121] Since the semantic domain network directly uses individual behavior labels as input, the learned semantic relationship between individual behavior and group behavior is the most superior, while the data domain network uses sensor data as input, and there is a more complex individual behavior feature learning process before the attention module. The learned attention weight is not as accurate as the semantic domain network. For this reason, this embodiment uses the attention knowledge of the semantic domain network to guide the attention of the data domain network. The semantic domain network and the data domain network are jointly trained, so that the data domain network can learn the entire optimization process of the semantic domain network, rather than just a final optimized result. The time back propagation algorithm is used to learn all the parameters of the data domain network to achieve the purpose of semantically guided data training. The semantic domain network only guides the data domain network in the training stage.

[0122] It should be noted that when training the network model, the training sample data are labeled, including individual behavior labels and group behavior labels, so that they can be used to calculate the loss function, which will not be repeated here.

[0123] Step S3: Use the trained data domain network to process the local position sensor data of each individual in the group to be detected, and predict the group behavior recognition result corresponding to the group to be detected.

[0124] When it is necessary to detect group behavior of a group, the trained data domain network can be used for detection.

[0125] It is only necessary to obtain the local position sensor data of each individual in the group to be detected, input it into the trained data domain network, and output the final group behavior recognition result.

[0126] Specifically, the group behavior features are used as the input of the fully connected layer, and the group behavior recognition results are obtained through the fully connected layer.

[0127] This application conducted experiments on group behavior recognition on a group behavior dataset constructed from the UT-Data dataset. The effectiveness of this method was verified in two cases. The first case was a fixed group of 5 people, in which one or two random individuals did not belong to the group, simulating individual non-critical behaviors in the group. The second group had a random number of people, but included a non-group individual. The recognition performance was measured by the mean ± standard deviation of the accuracy, precision, recall, and F1 score repeated 10 times under a five-fold cross validation. The experimental results are shown in Table 1:

[0128]

[0129] Table 1

[0130] Through the comparison of the three methods in two test cases, when the input contains non-critical behaviors, the more non-critical behaviors the data domain method contains in the input, the worse the recognition effect is, indicating that the data domain network cannot handle non-critical behaviors in the group well and cannot effectively suppress non-critical behaviors in the input data. This method can handle non-critical behaviors well even if the input contains non-critical behaviors. The recognition effect is better than that of a single semantic domain network and a data domain network. In the two test cases, the accuracy is improved by 25.3% and 27.7% respectively compared with the data domain method, which fully proves that this method can effectively suppress the interference caused by non-critical behaviors in the group. In addition, when the input contains fewer non-critical behaviors, it can be seen that the standard deviation is smaller and the volatility of each fold experimental result is smaller. The standard deviation of this method is the smallest in both test cases, and it has better robustness.

[0131] The above-mentioned embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the invention patent. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the patent of the present application shall be subject to the attached claims.

Claims

1. A sensor data group behavior recognition method based on joint attention mechanism, characterized in that: The sensor data group behavior recognition method based on the joint attention mechanism includes: Step 1: construct a network model including a data domain network and a semantic domain network, wherein the data domain network is used to process the local position sensor data of each individual in the group to obtain the data domain network group behavior characteristics, and then obtain the data domain network group behavior recognition result through a fully connected layer, and the semantic domain network is used to process the individual behavior label of each individual in the group to obtain the semantic domain network group behavior characteristics, and then obtain the semantic domain network group behavior recognition result through a fully connected layer; The data domain network is used to process the local position sensor data of each individual in the group to obtain the data domain network group behavior characteristics, including: For each individual, the continuous sensor data collected by the sensor at the individual's local position is segmented through a sliding window to obtain the individual's local position sensor data after segmentation, the individual's different local position sensor data are input into a convolutional neural network to obtain a first feature, and then the first feature is input into a bidirectional long short-term memory network to obtain a second feature, the first feature and the second feature are spliced ​​to obtain a personal local behavior feature, all different individual local behavior features are spliced ​​to obtain an individual behavior feature, and the individual behavior feature is passed through a fully connected layer to obtain an individual behavior recognition result; Calculate individual behavior correlation based on individual behavior characteristics, calculate individual position correlation based on individual coordinates, fuse individual behavior correlation and individual position correlation to obtain individual interaction relationship, input individual behavior characteristics and individual interaction relationship into the data domain graph convolution network, extract group interaction relationship characteristics, and concatenate group interaction relationship characteristics with individual behavior characteristics to obtain the third feature, which is then input into the attention module to learn and obtain data domain group behavior characteristics; The semantic domain network is used to process the individual behavior label of each individual in the group to obtain the semantic domain network group behavior characteristics, including: The individual behavior label is converted into a one-hot encoding vector, and the one-hot encoding vector and the individual position correlation are input into the semantic domain graph convolutional network to extract the semantic domain features. The semantic domain features are then input into the attention module to learn the semantic domain network group behavior features. Step 2: Construct a loss function to combine the data domain network and the semantic domain network, and perform joint training to obtain the network parameters of the data domain network; The loss function formula is as follows: Among them, L represents the joint loss, L p represents the classification loss of individual behavior recognition in the data domain network, L g represents the classification loss of group behavior recognition in the data domain network, L SPA represents the Euclidean distance between the data domain network and the semantic domain network, L KD represents the knowledge distillation loss, p i Represents individual behavior labels, represents the individual behavior recognition result of the data domain network, g i represents the group behavior label, P d represents the result of group behavior recognition in the data domain network, λ1 and λ2 are hyperparameters, represents the attention weight of the semantic domain network, represents the data domain network attention weight, P s Represents the result of semantic domain network group behavior recognition; Step 3: Use the trained data domain network to process the local position sensor data of each individual in the group to be detected, and predict the group behavior recognition result corresponding to the group to be detected.

2. The sensor data group behavior recognition method based on joint attention mechanism according to claim 1 is characterized in that: The calculating of individual behavior correlation based on individual behavior characteristics includes: The covariance between individual behavioral characteristics is used as a measure of the correlation between individual behaviors.

3. The sensor data group behavior recognition method based on joint attention mechanism according to claim 1 is characterized in that: The calculating of individual position correlation based on individual coordinates includes: Calculate the Euclidean distance between two objects. The formula is as follows: Among them, d ij (t) represents the Euclidean distance between individuals i and j at time t, and They represent the coordinates of individual i at time t respectively; The average value of the Euclidean distance between individuals i and j in the kth sliding window is expressed as The position correlation between individuals i and j in the kth sliding window is calculated by the following formula:

4. The sensor data group behavior recognition method based on joint attention mechanism according to claim 3 is characterized in that: The individual interaction relationship is obtained by fusing the individual behavior correlation and the individual position correlation, including: Individual interaction value The fusion formula is as follows: Among them, λ is the weight parameter, represents an individual behavior correlation measure value, where the individual behavior correlation measure value is the covariance between individual behavior features.

5. The sensor data group behavior recognition method based on joint attention mechanism according to claim 1 is characterized in that: The individual behavior characteristics and individual interaction relationships are input into the data domain graph convolutional network, wherein the individual behavior characteristics are used as nodes and the individual interaction relationships are used as edges between nodes.

6. The sensor data group behavior recognition method based on joint attention mechanism according to claim 1 is characterized in that: The one-hot encoding vector and the individual position correlation are input into the semantic domain graph convolutional network to extract semantic domain features, wherein the one-hot encoding vector is used as a node and the individual position correlation is used as an edge between nodes.

Images