Human behavior recognition method based on dual-channel mixed graph convolutional network

Abstract

The present application relates to the technical field of human behavior recognition, and particularly relates to a human behavior recognition method of a dual-channel mixed graph convolution network, comprising data preprocessing; a graph convolution module construction based on feature similarity; a graph convolution module construction based on individual characteristic difference; adding a dual-channel fusion based on self-attention to construct a human behavior recognition model of a dual-channel mixed graph convolution network based on feature similarity and individual characteristics. The present application is complementary to each other through the similarity between action features and the difference between individual actions, and can be adaptively fused to obtain more deeply related information for the classification task.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of human behavior recognition, and particularly relates to a human behavior recognition method based on a dual-channel mixed graph convolution network. BACKGROUND

[0002] Human activity recognition (HAR) based on sensors is an important field of human behavior perception, which is widely used in behavior analysis, healthcare, remote monitoring of the elderly, smart home, campus bullying detection, etc. This provides multi-dimensional support for the quality of life and safety of human beings.

[0003] HAR recognition based on machine learning has achieved good results in the last decade. However, the machine learning-based solution completely depends on the pre-processing result of the original data. In other words, the more significant the features extracted by pre-processing, the more prominent the classification result. This challenge can be easily solved or overcome by a deep learning model, and has better generalization performance and robustness than machine learning. For this reason, many researchers encode the original time series signal into an image, and recognize human activities through a convolutional neural network (CNN). However, these models focus more on feature extraction and classification of a single sample, ignoring the potential information existing in the context of the time series signal. Or through a long short-term memory network (LSTM) to capture the temporal context to improve the classification performance, but they ignore the potential relationship information between the same actions.

[0004] Unlike video or time series signals, graph structure data in a non-Euclidean space contains rich relationship information. Graph neural networks (GNNs) have become the focus of deep learning due to their strong reasoning ability and good interpretability. At present, few researchers use GNNs to learn and mine the topological relationship hidden in human activity recognition based on wearable sensor systems.

[0005] Starting from life, different people have different actions of the same kind (intra-class difference), and the same person has different actions (inter-class similarity). These intra-class differences and inter-class similarities may come from the environment, disease, habit, etc. When the sample attributes are uniformly extracted in all element dimensions, the intra-class difference becomes the main reason for the misclassification of actions. It is very difficult to recognize these hidden information, but it fully contains the habit characteristics of individuals. SUMMARY

[0006] In view of the shortcomings of the existing algorithm: the present application is complementary to the similarity between action features and the difference between individual actions, and can be adaptively fused to obtain more deeply related information for the classification task.

[0007] The technical solution adopted in this invention is: a human behavior recognition method using a dual-channel hybrid graph convolutional network, comprising the following steps:

[0008] Step 1: Data preprocessing;

[0009] Furthermore, specifically including:

[0010] The three-axis accelerations of the accelerometer and gyroscope are calculated respectively, and the samples are divided according to a fixed-width sliding window with a preset step size and overlap rate.

[0011] The time-frequency domain features of the time series data in the sample are extracted using the discrete wavelet packet transform algorithm.

[0012] Furthermore, the formula for the discrete wavelet packet transform algorithm is as follows:

[0013]

[0014] in, These are wavelet packet decomposition coefficients. These are the coefficients of the high-pass filter. These are the coefficients of the low-pass filter. and It represents the number of nodes in the wavelet packet.

[0015] Step 2: Constructing a graph convolution module based on feature similarity;

[0016] Furthermore, specifically including:

[0017] Step 21, adopt k Nearest neighbor algorithm constructs graph Adjacency matrix of action feature maps Calculate the pairwise Euclidean distance between all action nodes. Then on Sort by size from smallest to largest and select the nearest one. Each node serves as a neighbor node;

[0018] Select the most recent The formula for a node to be a neighbor node is:

[0019]

[0020] in, Represents a node Euclidean distances between the node and other nodes It is an ascending order operator. It is sorted , yes Center front The number of node numbers.

[0021] Further, the adjacency matrix formula of the action feature map is:

[0022]

[0023] wherein, represents whether there is an edge connection relationship between the th node and the th node.

[0024] Step 22, taking the feature map as an input graph into the similarity formula of human behavior to perform spectral convolution operation to obtain a new node representation;

[0025] Further, the similarity formula of human behavior is:

[0026]

[0027] wherein, is an activation function, is a weight matrix of the action common graph convolution layer, is the output of the layer and ; indicates that when performing graph convolution, the node can learn more about its own features; is a diagonal degree matrix, and the diagonal elements .

[0028] Step three, constructing a graph convolution module based on individual characteristic differences;

[0029] Further, it specifically includes:

[0030] Step 31, constructing an individual characteristic graph;

[0031] Further, it specifically includes:

[0032] First, taking each movement sample as an entity node, and the corresponding wavelet packet transform feature as the node feature ;

[0033] Then, using an undirected edge to connect two consecutive time periods, and obtaining an individual characteristic graph through the adjacent matrix of the individual characteristic graph;

[0034] The adjacent matrix of the individual characteristic graph is:

[0035]

[0036] wherein, represents the serial number of the node, a sequence number of the node, a sequence number of the node a time sequence data starting sampling point of the node, a sequence number of the node a time sequence data starting sampling point of the node.

[0037] Step 32, using the personal characteristic graph as an input graph, the first layer output is represented as formula (7):

[0038]

[0039] wherein, is an activation function, is a weight matrix of the action common graph convolution first layer, is the output of the first layer and ; indicates that the node can learn more about its own features when performing graph convolution; is a diagonal degree matrix, and the diagonal elements .

[0040] Step four, adding a double-channel fusion based on self-attention to construct a human behavior recognition model of a double-channel mixed graph convolution network based on feature similarity and personal characteristics;

[0041] Further, specifically comprising:

[0042] The importance of the embedding obtained by the graph convolution in the two topological graphs is adaptively obtained through the importance formula;

[0043] The learned importance is converted into a diagonal matrix, and finally the embedding value related to the final classification task is obtained through the embedding formula.

[0044] Further, the importance formula is:

[0045]

[0046] wherein, , respectively represent the attention values of the N nodes of the feature , ; , is a weight matrix, is a bias vector;

[0047] The embedding formula is:

[0048]

[0049] wherein, , is a weight , is converted into a diagonal matrix.

[0050] Advantages of the present application:

[0051] Firstly, the similarity between the same actions is calculated to construct a feature graph; the wavelet coefficients of the sampling sequence are used as the node features in the feature graph, and the nodes with strong correlation are connected to each other, and the features between the neighbors are aggregated by GCNs to strengthen the connection between the same actions; however, there are differences between the same category actions of each person; if only the similarity of the intra-class actions is used to construct the graph, the individual differences existing in the graph will become interference information between the node features; in order to solve this limitation, a personal feature graph with linear structure is constructed by collecting the individual actions in the same period according to the time stamp; the node features in the personal feature graph are the same as those in the feature graph, but the nodes are connected to each other only when and if the time stamps of the two nodes are continuous, and the features between the neighbors are aggregated by GCNs to make the individual difference information of the actions become the identification information of the action individuals in the graph; finally, in order to better fuse the features extracted by the two convolutional networks, an attention mechanism is added to learn the importance weight between different features. BRIEF DESCRIPTION OF DRAWINGS

[0052] Figure 1 is a flow chart of the human behavior recognition method of the dual-channel hybrid graph convolutional network of the present application;

[0053] Figure 2 is a wavelet decomposition structure diagram of the present application;

[0054] Figure 3 is a feature graph constructed by some action samples in the UCI-HAR data set;

[0055] Figure 4 is a graph convolution module based on feature similarity of the present application;

[0056] Figure 5 is a personal feature graph constructed by the action data of two different persons in the UCI-HAR data set;

[0057] Figure 6 is a graph convolution module based on the personal feature graph;

[0058] Figure 7 is a human behavior recognition model framework of the dual-channel hybrid graph convolutional network based on feature similarity and personal features. DETAILED DESCRIPTION

[0059] The application will be further described below in connection with the accompanying drawings and examples, which are simplified schematic diagrams and only schematically illustrate the basic structure of the application, and thus only show the components related to the application.

[0060] As shown in the drawings, Figure 1 A human behavior recognition method of a double-channel hybrid graph convolutional network includes the following steps:

[0061] Step 1, data preprocessing;

[0062] In the present application, the performance of the model is evaluated by the data disclosed in UCI-HAR;

[0063] The data preprocessing step includes:

[0064] Step 11, window division: the data collected by the acceleration sensor is a three-axis acceleration signal, in order to weaken the directional error caused by the installation of the sensor, the three-axis acceleration signal is subjected to amplitude scalar operation to synthesize a three-axis combined acceleration signal, and formula (1) is the amplitude calculation process of the combined acceleration:

[0065] (1)

[0066] In the formula, represents the acceleration signal in the front-back direction, represents the acceleration signal in the left-right direction, represents the acceleration signal in the up-down direction, and X , Y , Z and the three-axis combined acceleration are subjected to window division processing.

[0067] Step 12, discrete wavelet packet transform: discrete wavelet transform can describe the local characteristics of time series signals in time and frequency domains, and is an effective signal analysis technology; and discrete wavelet packet transform is an improvement on wavelet transform, which further decomposes the high and low frequency subbands at each level of signal decomposition based on wavelet transform, and finally calculates the optimal signal decomposition path by minimizing a cost function; intuitively, the data format of the input segment is wherein, is the number of channels of the data; this means that two sensors are used, each sensor has 4 channels (X, Y, Z, three-axis and acceleration), and there will be an input matrix of ; and based on DWPT, each row of motion time series signal is decomposed, and the wavelet packet transform is shown in formula (2).

[0068] (2)

[0069] wherein, is a wavelet packet decomposition coefficient, is a high-pass filter coefficient, is a low-pass filter coefficient, and is a wavelet packet node number.

[0070] The performance of the method of the present application is evaluated using the data disclosed in UCI-HAR; first, the three-axis and acceleration of the accelerometer and gyroscope are calculated by formula (1) respectively, and the sample is divided into a fixed width sliding window (128 readings, window) with 2.56s and 50% overlap.

[0071] Secondly, the time-frequency domain feature extraction is performed on the 8 time series data in the sample by the discrete wavelet packet transform formula (2). Figure 2 The three-layer wavelet packet decomposition structure diagram of the original data of the accelerometer and gyroscope sensor of a sample is shown in the figure, the left side is the wavelet packet transform of the accelerometer, and the right side is the gyroscope. Among them, each sensor has four channels; the db3 mother wavelet is used to decompose the motion signal of each channel to obtain eight sub-bands from left to right with high to low frequency, and the sub-band length is calculated by two norm; 8 time-frequency domain features of a single channel of the sensor are obtained. For a three-axis sensor, wavelet transform is performed on the X-axis, Y-axis, Z-axis and three-axis and acceleration respectively to obtain 32 time-frequency domain features; the present application uses the sensor data collected by the three-axis accelerometer and three-axis gyroscope, so 64-dimensional features are extracted from the sensor data under a window, i.e. sample .

[0072] As Figure 4 , step two, the graph convolution module is constructed based on feature similarity;

[0073] Specifically, it includes:

[0074] Step 21, feature map construction, in order to capture the similarity information between the same type of action, a feature map with intra-class similarity is constructed based on the node feature matrix -neighborhood . , wherein, is the adjacency matrix of the action feature map; specifically, the smaller the distance between two nodes, the more likely it is to have a similarity property and thus interact. Then, first calculate the Euclidean distance between all action nodes ; then sort from small to large, and select the nearest nodes as neighbor nodes, as formula (3):

[0075] (3) ​

[0076] wherein, denotes a node the Euclidean distance between the node and other nodes, is an ascending operator, is the sorted , is the node sequence number of the first number of nodes.

[0077] adjacency matrix of the action feature graph is defined as formula (4):

[0078] (4)

[0079] wherein, denotes whether there is an edge connection relationship between the th node and the th node, if the two nodes are connected.

[0080] Each movement sample is taken as an entity node, and the corresponding 64 wavelet packet transform features are taken as node features ; the k-neighbor algorithm is used to construct the adjacency matrix k of the graph , the Euclidean distance between the node features of the patient samples is calculated by formula (3) , the correlation between the nodes is obtained ; then, the are sorted from small to large, the nearest nodes are selected as neighbor nodes, and the adjacency matrix of the feature graph such as is obtained by formula (4) Figure 3 .

[0081] Step 22, using the action feature graph as the input graph, the output of the first layer can be represented as formula (5):

[0082] (5)

[0083] wherein, is an activation function, is the weight matrix of the action common graph convolution first layer, is the output of the first layer and ; indicates that when the graph convolution is performed, the node can learn more about its own features; ​​is a diagonal matrix whose diagonal elements ; similarity information of human behavior can be captured in behavior graphs with intra-class similarity .

[0084] The constructed feature map is taken as an input graph into equation (5) for spectral convolution operation, and a new node representation can be obtained.

[0085] As Figure 6 shown, step three, a graph convolution module is constructed based on individual characteristic differences;

[0086] Specifically, it includes:

[0087] Step 31, individual characteristic graph construction;

[0088] After the data is segmented and wavelet decomposition is completed, is passed to the behavior individual characteristic graph construction module, which converts unstructured raw data into structured graph topology network data; each sample point is regarded as a node, the wavelet transform result corresponding to the sample point is a feature, then two consecutive time periods are connected using undirected edges, and the idea of connecting nodes with nodes comes from the fact that two consecutive time periods of human activity data are logically connected in time series, and the previous action will generate the next action under the influence of human habits; therefore, the degrees of the first and last time period nodes of the individual characteristic graph are 1, and the degrees of the remaining nodes are 2, and the adjacency matrix of the individual characteristic graph is given as formula (6):

[0089] (6)

[0090] wherein, denotes the node number , denotes the node number , denotes the time series data starting sampling point of the node , denotes the time series data starting sampling point of the node, and ( is the number of sampling points).

[0091] Each motion sample is regarded as an entity node, and the corresponding 64 wavelet packet transform features are regarded as node features ​; then use undirected edges to connect two consecutive time periods, the idea of connecting between nodes comes from the fact that two consecutive time periods in the time series are logically connected, and the previous action will generate the next action under the influence of human habits. Therefore, the degree of the first and last time period nodes of the personal characteristic graph is 1, and the degree of the remaining nodes is 2, and the personal characteristic graph is obtained by formula (6) Figure 5 adjacency matrix .

[0092] Step 32, constructing a graph convolution module based on personal behavior characteristics;

[0093] Unlike the graph convolution module based on feature similarity, the graph convolution module based on personal characteristic graph focuses on the message passing between individual actions in a period of time; using the personal characteristic graph As the input graph, the first layer output Can be represented as formula (7):

[0094] (7)

[0095] Wherein, is the activation function, is the weight matrix of the action commonality graph convolution layer, is the output of the layer and ; Indicates that when performing graph convolution, the node can learn more about its own features; is a diagonal degree matrix, and the diagonal elements .

[0096] Step four, adding a double-channel fusion based on self-attention to construct a human behavior recognition model based on a double-channel mixed graph convolution network based on feature similarity and personal characteristics, as shown in Figure 7 .

[0097] Specifically, it includes:

[0098] The center node performs message passing with similar nodes in the feature graph; when the center node is connected with more similar nodes, the center node can obtain information most relevant to the classification task; however, some nodes have low similarity with nodes of the same category, and the center node is difficult to obtain information most relevant to the classification task, resulting in classification errors of these nodes; in this case, it is likely that the action difference information in the center node becomes interference information of the node; at this time, the personal feature graph can be used to convert the difference information into habit information of the individual to improve the classification effect; of course, in real life, the conversion of human actions is very rapid, and the probability of the occurrence of the case that the center node is completely different from the surrounding nodes is very large, at this time, the classification of the center node needs to be completed by means of message passing between similar actions in the feature graph, and an attention mechanism such as formula (8) is used to learn the importance of them , so as to solve the needs of different nodes, and the importance formula is:

[0099] (8)

[0100] wherein, , respectively represent the attention values of N nodes of the feature , . , is a weight matrix, is a bias vector; the greater the corresponding , the more important the corresponding embedding in the structure graph where the node is located; for all N nodes, the weight , has been learned and converted into a diagonal matrix , , so as to obtain the final embedding , as shown in formula (9):

[0101] (9)

[0102] First, the importance of the embedding obtained by the graph convolution in the two topological graphs is adaptively obtained by formula (8), and then the learned importance is converted into a diagonal matrix, and finally the embedding related to the final classification task is obtained by formula (9) .

[0103] Based on the above ideal embodiments according to the present application, through the above description, relevant personnel can make various changes and modifications without deviating from the technical idea of the present application. The technical scope of the present application is not limited to the contents in the specification, and the technical scope must be determined according to the scope of claims.

Claims

1. A method for human behavior recognition using a dual-channel hybrid graph convolutional network, characterized in that, Includes the following steps: Step 1: Data preprocessing; Step 2: Constructing a graph convolution module based on feature similarity; Step two specifically includes: Step 21, adopt k Nearest neighbor algorithm constructs graph Adjacency matrix of action feature maps Calculate the pairwise Euclidean distance between all action nodes. Then on Sort by size from smallest to largest and select the nearest one. Each node is considered a neighbor node, and the formula is as follows: in, Represents a node Euclidean distances between the node and other nodes It is an ascending order operator. It is sorted , yes Center front The number of node indices; Step 22: Transfer the feature map The similarity formula of human behavior is used as the input image to perform spectral convolution operation to obtain a new node representation; The formula for similarity in human behavior is: in, It is an activation function. It is the first convolution of the action commonality graph. The weight matrix of the layer, It is the first The output of the layer and ; This means that nodes can learn more of their own features when performing graph convolution; A diagonal degree matrix, whose diagonal elements ; Step 3: Constructing a graph convolution module based on individual characteristics and differences; Step three specifically includes: Step 31: Constructing a Personal Feature Map; Step 32: Use a personal profile picture As input graphics, the first Layer output Represented as: in, It is an activation function. It is the first convolution of the action commonality graph. The weight matrix of the layer, It is the first The output of the layer and ; This indicates that nodes can learn more of their own features during graph convolution; This is a diagonal degree matrix, with diagonal elements. ; Step 4: Add self-attention-based dual-channel fusion to construct a human behavior recognition model based on feature similarity and personal characteristics using a dual-channel hybrid graph convolutional network; Step four specifically includes: The importance of the embedding is obtained by adaptively acquiring graph convolution in two topological graphs using an importance formula; The learned importance is then converted into a diagonal matrix, and finally the embedding values ​​related to the final classification task are obtained through the embedding formula.

2. The human behavior recognition method using a dual-channel hybrid graph convolutional network according to claim 1, characterized in that, Step one specifically includes: The three-axis accelerations of the accelerometer and gyroscope are calculated respectively, and the samples are divided according to a fixed-width sliding window with a preset step size and overlap rate. The time-frequency domain features of the time series data in the sample are extracted using the discrete wavelet packet transform algorithm.

3. The human behavior recognition method using a dual-channel hybrid graph convolutional network according to claim 2, characterized in that, The formula for the discrete wavelet packet transform algorithm is: in, These are wavelet packet decomposition coefficients. These are the coefficients of the high-pass filter. These are the coefficients of the low-pass filter. and It represents the number of nodes in the wavelet packet.

4. The human behavior recognition method using a dual-channel hybrid graph convolutional network according to claim 1, characterized in that, The adjacency matrix formula for the feature map is: in, Indicates the first The node and the first Does an edge connection exist between the nodes? 5. The human behavior recognition method using a dual-channel hybrid graph convolutional network according to claim 1, characterized in that, Step 31 specifically includes: First, each motion sample is treated as an entity node, and the corresponding wavelet packet transform features are used as node features. ; Then, two consecutive time periods are connected using undirected edges, and the adjacency matrix of the individual feature graph is used. Obtain a personal profile image; Adjacency matrix of personal feature map The formula is: in, Indicates the node's sequence number. Indicates the node's sequence number. Represents a node The starting sampling point of the time series data. express The starting sampling point of the node's time-series data.

6. The human behavior recognition method using a dual-channel hybrid graph convolutional network according to claim 1, characterized in that, The importance formula is: in, , Representing features respectively , The attention values ​​of N nodes; , It is a weight matrix. It is the bias vector; The embedding formula is: in, , For weight , Convert to a diagonal matrix.

Images