Human behavior recognition method and system based on multi-channel directed graph convolution

Abstract

The application provides a human behavior recognition method and system based on a multi-channel directed graph convolution, comprising: obtaining a human skeleton data sequence to be recognized, and performing corresponding preprocessing to obtain joint flow, skeleton flow and motion flow data; performing directed graph construction based on the preprocessed skeleton data to obtain a correlation matrix corresponding to the human skeleton data; inputting the joint flow, the skeleton flow and the motion flow into a pre-trained multi-channel directed graph convolution model respectively, using pre-set classifiers to obtain scores of different categories corresponding to different flows based on feature information output by the multi-channel directed graph convolution model, and adding the scores of the categories corresponding to different flows to take the category with the maximum score as a final human behavior recognition result.

Description

TECHNICAL FIELD

[0001] The present application belongs to the technical field of human behavior recognition, and particularly relates to a human behavior recognition method and system based on multi-channel directed graph convolution. BACKGROUND

[0002] The statements in this section merely provide background information related to the present application and do not necessarily constitute the prior art.

[0003] Human behavior recognition refers to a recognition task of recognizing the specific action of a person in a video by a specific algorithm from the features of the human body and the features of the action completed. In the field of computer vision, human behavior recognition is a hot issue that has been widely concerned, and has been widely used in human-computer interaction, intelligent security, medical health and sports analysis and other fields. The basic process of the behavior recognition task is generally as follows: data image preprocessing, human body detection in motion, motion feature extraction, feature training and classification, and behavior recognition. However, the traditional recognition method is based on an RGB data set, and such data is easily affected by environmental factors including background and light. Human skeleton joint data is a high-level feature of the human body, is almost not affected by appearance, can better avoid the noise influence caused by background, light and perspective changes, has good robustness, and is also very effective in terms of calculation and storage.

[0004] Therefore, the research on the human behavior recognition method based on skeleton data not only has important significance in theoretical academic research, but also has great value in practical application. The method mentioned in this paper is based on a skeleton data set.

[0005] The input of the human behavior recognition algorithm based on skeleton data is usually a set of coordinates of a series of points, which represent the positions of the skeleton joints. The common deep learning method for processing skeleton joints mainly includes three types: convolutional neural network (CNN), recurrent neural network (RNN) and graph convolutional network (GCN). Since the inputs of different deep learning networks are different, the representation methods of the joint data are also different. The input of the convolutional neural network is a structured picture, and the joint data is represented by a pseudo-image; the input of the recurrent neural network is a data stream connected to each other, and the joint data is represented by a vector sequence; the input of the graph convolutional network is a graph structure, and the joint data is represented by a topological graph.

[0006] However, the inventors find that the "pseudo image" and the "vector sequence" mentioned above are both data based on Euclidean space. In the field of feature extraction of rule-based data based on Euclidean space, both convolutional neural networks and recurrent neural networks have achieved great success. However, in the field of human behavior recognition based on skeleton data, the focus of our attention is the human skeleton joint data on the video clip, and we also perform feature extraction on the skeleton joint data. However, the skeleton joint is irregular in internal structure and is data based on non-Euclidean space. We call this structure a graph structure, also known as a topological structure. The distribution of other nodes near each node in the graph structure may be unique, and the distribution of each node is uncertain. This is the biggest difference between the graph structure and the "two-dimensional picture" mentioned above, and it does not have translational invariance. The ability of convolutional neural networks or recurrent neural networks to extract features from such structured data is greatly reduced, which greatly reduces the accuracy of traditional human behavior recognition algorithms based on convolutional neural networks and recurrent neural networks. SUMMARY

[0007] To solve the above problems, the application provides a human behavior recognition method and system based on multi-channel directed graph convolution. The scheme extracts features of human behavior based on a proposed multi-channel directed graph convolution model. The scheme combines a multi-channel directed spatial convolution module and a multi-scale contrastive temporal convolution. The channel-specific global and local features are obtained through the multi-channel directed spatial convolution module. Then, the two features are fused, and the fused data is input into the multi-scale contrastive temporal convolution module to extract rich time domain information and obtain multi-scale time features. The scheme can effectively aggregate motion information between different frames and effectively improve the accuracy of behavior recognition.

[0008] According to a first aspect of an embodiment of the application, a human behavior recognition method based on multi-channel directed graph convolution is provided, comprising:

[0009] Obtaining a human skeleton data sequence to be recognized and performing corresponding preprocessing to obtain joint flow, skeleton flow and motion flow data;

[0010] Based on the preprocessed skeleton data, a directed graph is constructed to obtain a correlation matrix corresponding to the human skeleton data;

[0011] input the joint flow, the bone flow and the motion flow into a pre-trained multi-channel directed graph convolution model respectively, wherein the multi-channel directed graph convolution model comprises a plurality of sequentially connected spatio-temporal convolution blocks, each spatio-temporal convolution block comprises a plurality of sequentially connected multi-channel directed spatial convolution modules for extracting spatial information and aggregation and a multi-scale temporal convolution module for obtaining time domain information, and each spatio-temporal convolution block except the first layer further comprises a processing process of merging the feature information of the output data of the previous spatio-temporal convolution block after residual convolution processing and the output of the multi-scale temporal convolution module; in the multi-channel directed spatial convolution module, the correlation matrix is used as the shared topology of all channels, and the topology of each channel is modeled by taking the structure of the shared topology as prior knowledge;

[0012] Based on the feature information output by the multi-channel directed graph convolution model, a pre-set classifier is used to obtain scores of different categories corresponding to different flows, and the category with the maximum score is taken as the final human behavior recognition result by adding the scores of the categories corresponding to different flows.

[0013] Further, in each spatio-temporal convolution block, the input data passes through the multi-channel directed spatial convolution module to obtain channel-specific global features and local features, and the global features and the local features are fused and input into the multi-scale temporal convolution module to obtain multi-scale time features.

[0014] Further, the multi-scale time module comprises a first branch and a second branch arranged side by side for extracting multi-scale time features, a third branch for removing redundant information in sequence frames, a fourth branch for processing current frame features, and a fifth branch for obtaining non-local features.

[0015] Further, the output of the fifth branch in the multi-scale time module is compared with the output of the other branches respectively, and the contrast learning is used to maximize the different features between frames and enhance the feature representation.

[0016] Further, the multi-channel directed spatial convolution module comprises a plurality of channel-specific topology branches and a directed edge convolution branch, the channel-specific topology branch first infers the topology structure specific to each channel by using dynamic topology, then models the non-naturally connected joints by global attention graph convolution, and extracts the global feature-specific topology structure on the channel by an update function; the directed edge convolution branch aggregates local features by constructing a plurality of local graphs in the skeleton graph.

[0017] Further, the directed graph is constructed based on the pre-processed skeleton data, specifically: taking the bone as the edge of the directed graph and the joint as the vertex of the directed graph, and each directed edge is pointed from the vertex close to the skeleton center of gravity to the vertex far from the skeleton center of gravity.

[0018] Further, the pre-set classifier adopts a full connection layer.

[0019] According to a second aspect of the embodiments of the present application, a human behavior recognition system based on multi-channel directed graph convolution is provided, comprising:

[0020] a data acquisition unit configured to acquire a human skeleton data sequence to be recognized, and perform corresponding preprocessing to obtain joint flow, skeleton flow and motion flow data;

[0021] a directed graph construction unit configured to construct a directed graph based on the preprocessed skeleton data to obtain a correlation matrix corresponding to the human skeleton data;

[0022] a feature extraction unit configured to input the joint flow, the skeleton flow and the motion flow into a pre-trained multi-channel directed graph convolution model respectively, wherein the multi-channel directed graph convolution model comprises a plurality of sequentially connected spatio-temporal convolution blocks, each spatio-temporal convolution block comprises a plurality of sequentially connected multi-channel directed spatial convolution modules for extracting spatial information and aggregation, and a multi-scale temporal convolution module for obtaining time domain information, and in addition to the first layer, each spatio-temporal convolution block further comprises a processing process of merging the feature information of the output data of the previous spatio-temporal convolution block after residual convolution processing and the output of the multi-scale temporal convolution module; in the multi-channel directed spatial convolution module, the correlation matrix is used as a shared topology of all channels, and the topology of each channel is modeled by taking the structure of the shared topology as prior knowledge;

[0023] a classification and recognition unit configured to obtain scores of different categories corresponding to different flows based on the feature information output by the multi-channel directed graph convolution model, using a pre-set classifier, and add the scores of the categories corresponding to different flows to take the category with the maximum score as the final human behavior recognition result.

[0024] According to a third aspect of the embodiments of the present application, an electronic device is provided, comprising a memory, a processor and a computer program stored on the memory and running on the memory, wherein the processor implements the human behavior recognition method based on multi-channel directed graph convolution when executing the program.

[0025] According to a fourth aspect of the embodiments of the present application, a non-transitory computer readable storage medium is provided, which stores a computer program, and the program is executed by a processor to implement the human behavior recognition method based on multi-channel directed graph convolution.

[0026] Compared with the prior art, the present application has the following advantages:

[0027] (1) The application provides a human behavior recognition method and system based on multi-channel directed graph convolution, which extracts the features of human behavior based on the proposed multi-channel directed graph convolution model.

[0028] (2) In the scheme, contrastive learning is introduced between different scale time features in the multi-scale contrastive time convolution module to effectively enhance the communication between different features, enhance the feature representation, and further improve the feature description effect.

[0029] (3) In the scheme, a channel-specific topology branch is designed in the multi-channel directed spatial convolution module, which discards static topology (i.e. aggregation of the same topology in different channels) and selects dynamic topology to improve the flexibility of the method. In this branch, the topology structure specific to each channel is first inferred, and then global attention graph convolution is used to better model the non-natural connected joints and extract the global feature specific to the topology structure on the channel. At the same time, as a supplement, a directed edge convolution branch is added in the multi-channel directed spatial convolution module to extract deeper local features that cannot be captured by the channel-specific topology branch.

[0030] The advantages of the additional aspects of the application will be partially given in the following description, partially will become apparent from the following description, or will be understood by the practice of the application. BRIEF DESCRIPTION OF DRAWINGS

[0031] The drawings accompanying the specification of the application are used to provide a further understanding of the application, the illustrative embodiments of the application and their description serve to explain the application, and do not constitute an improper limitation of the application.

[0032] Figure 1 The general skeleton and directed skeleton diagram described in the embodiments of the application are shown;

[0033] Figure 2 The flowchart of the behavior recognition method based on multi-channel directed graph convolution described in the embodiments of the application is shown;

[0034] Figure 3 The architecture diagram of each layer of basic spatio-temporal convolution block in the behavior recognition method based on multi-channel directed graph convolution described in the embodiments of the application is shown;

[0035] Figure 4The architecture diagram of the channel-specific topology modeling module described in the embodiments of the present application. DETAILED DESCRIPTION

[0036] The present application is further described below in conjunction with the accompanying drawings and embodiments.

[0037] It should be noted that the following detailed description is illustrative only and is intended to provide further description of the present application. Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which the present application belongs.

[0038] It should be noted that the terms used herein are only intended to describe specific embodiments and are not intended to limit exemplary embodiments according to the present application. As used herein, the singular form is intended to include the plural form unless the context clearly indicates otherwise, and it should also be understood that when the terms "comprise" and / or "include" are used in the specification, there is a presence of the features, steps, operations, devices, components and / or combinations thereof.

[0039] The embodiments in the present application and the features in the embodiments can be combined with each other without conflict.

[0040] Embodiment one:

[0041] The purpose of this embodiment is to provide a human behavior recognition method based on multi-channel directed graph convolution.

[0042] A human behavior recognition method based on multi-channel directed graph convolution, comprising:

[0043] Obtain the human skeleton data sequence to be recognized, and perform corresponding preprocessing to obtain joint flow, skeleton flow and motion flow data;

[0044] Based on the preprocessed skeleton data, a directed graph is constructed to obtain a correlation matrix corresponding to the human skeleton data;

[0045] The joint flow, skeleton flow and motion flow are respectively input into a pre-trained multi-channel directed graph convolution model, wherein the multi-channel directed graph convolution model comprises a plurality of sequentially connected spatio-temporal convolution blocks, each spatio-temporal convolution block comprises a plurality of sequentially connected multi-channel directed spatial convolution modules for extracting spatial information and aggregation, and a multi-scale time convolution module for obtaining time domain information, and in addition to the first layer, each spatio-temporal convolution block further comprises a processing process of merging the feature information of the output data of the previous spatio-temporal convolution block after residual convolution processing and the output of the multi-scale time convolution module; in the multi-channel directed spatial convolution module, the correlation matrix is used as the shared topology of all channels, and the topology of each channel is modeled by taking the structure of the shared topology as prior knowledge.

[0046] Based on the feature information output by the multi-channel directed graph convolution model, a pre-set classifier is used to obtain scores of different classes corresponding to different flows, and the class with the maximum score is taken as the final human behavior recognition result by adding the scores of different classes corresponding to different flows.

[0047] Unlike other methods, the method proposed in the embodiment represents the skeleton as a directed graph, and the edges and vertices represent the skeleton and joints, respectively. Each directed edge points from a vertex close to the center of mass of the skeleton to a vertex far from the center of mass of the skeleton. Moreover, we add a self-loop to the vertex representing the center of mass of the skeleton to keep the number of joints and skeletons consistent.

[0048] In the embodiment, the multi-channel directed graph convolution-based behavior recognition method includes joint flow, skeleton flow and motion flow, and each flow has ten basic spatio-temporal convolution blocks. Each spatio-temporal convolution block has a multi-channel directed spatial convolution module, a multi-scale contrastive temporal convolution module and a residual connection, and it is worth noting that the first layer of the spatio-temporal convolution block does not have a residual connection.

[0049] In each spatio-temporal convolution block, the data is first input into the multi-channel directed spatial convolution module to obtain channel-specific global features and local features, and then the two kinds of features are fused. We input the fused data into the multi-scale contrastive temporal convolution module to extract rich time domain information to obtain multi-scale time features. In addition, we introduce contrastive learning between time features of different scales to enhance feature exchange. Then, after passing through the activation function, the data is sent to the next layer of the basic spatio-temporal convolution block.

[0050] In the embodiment, a channel-specific topology branch is designed in the multi-channel directed spatial convolution module, which discards the static topology (i.e., the aggregation of the same topology in different channels) and selects the dynamic topology to improve the flexibility of the method. In this branch, the topology structure specific to each channel is first inferred, and then a global attention graph convolution is proposed to better model the joints with unnatural connections and extract the global feature-specific topology structure on the channel. In addition, a directed edge convolution branch is added in the multi-channel directed spatial convolution module to extract deeper local features that cannot be captured by the channel-specific topology branch. The directed edge convolution branch aggregates local features by constructing multiple local graphs in the skeleton graph to maintain the local structure. In order to model actions with different durations, multi-scale contrastive temporal convolution blocks are added in the method to process time domain information from the multi-channel directed spatial convolution module, which contains 5 branches. The non-local branch obtains long skip connection features and aggregates it with features extracted from other branches using contrastive learning. After training through the layer-by-layer network, global average pooling is performed to prevent overfitting, and finally a fully connected layer is used for classification operation.

[0051] Specifically, for ease of understanding, the scheme described in the present embodiment is described in detail from the perspective of specific implementation in combination with the drawings as follows:

[0052] Step one: preprocessing of skeleton data;

[0053] In actual application process, human skeleton information is easy to obtain through hardware device (such as kinetics) or software (openpose), which will not be repeated here. For convenience, the dataset used in the present embodiment is NTU-RGB+D60 / 120 dataset, which is composed of many text files, each file containing frame number of skeleton data, number of people performing action, (x, y, z) coordinates of each joint, etc. The preprocessing part of the dataset is to encapsulate the text data into a 4-dimensional matrix format of (C, T, V, M) so that it can be input into the network, where C represents the number of channels, T represents the number of frames, V represents the number of nodes, and M represents the number of moving people. This is the input data of joint flow. Then the input data of skeletal flow and motion flow are calculated through the formula, and are input into the corresponding flow.

[0054] The preprocessing of skeleton data is to extract the specific information required for network training, such as skeletal point coordinates, frame length, and joint node number. In specific implementation, it is encapsulated into a format that can be input into the network by using the Dataset and Dataloader modules provided by Pytorch, that is, a five-dimensional vector of (N, C, T, V, M). N, C, T, V, and M represent the number of batch processing, the number of channels, the number of frames, the number of nodes, and the number of moving people in a frame, respectively. This is the input data of joint flow. Then the input data of skeletal flow and motion flow are calculated through the formula e 1，2 =(x1-x2, y1-y2, z1-z2) and the formula m t =(x t+1 -x t , y t+1 -y t , z t+1 -z t ), and are input into the corresponding flow.

[0055] Step two: construction of directed graph;

[0056] This step represents the skeleton data as a directed acyclic graph, where the joints are vertices and the skeletons are edges. The direction of each edge is determined by the distance between the vertices and the center of gravity of the skeleton, where the vertex closer to the center of gravity points to the vertex farther from the center of gravity. The adjacency matrix is used to represent the direction of the skeleton between two adjacent joints. Given a directed graph with V vertices and E edges, the adjacency matrix is a V x E matrix A. The adjacency matrix composed of source vertices is called matrix A sThe incidence matrix consisting of the target vertices is called matrix A t The three matrices are combined together to form a joint matrix with size (3, V, E) input into the network.

[0057] In a specific implementation, the direction of the bone between two adjacent joints is represented by using the incidence matrix. In order to keep the number of joints and bones consistent, a self-loop is added to the vertex representing the center of gravity of the skeleton. Given a directed graph with V vertices and E edges, the incidence matrix A of A is a V x E matrix, and the element (for example: a ij = 1, i = 1,..., V, j = 1,..., E) represents the relationship between the corresponding vertex (v i ) and edge (e j ). If v i is the source vertex of e j , or e j is the outgoing edge of v i , then a ij = -1; if v i is the target vertex of e j , or e j is the incoming edge of v i , then a ij = 1; if there is no connection, it is 0. In order to distinguish the source vertex and the target vertex, the incidence matrix consisting of the source vertices is called matrix A s , and the incidence matrix consisting of the target vertices is called matrix A t . A s is the absolute value of the position taking -1 in the matrix A, and A t is the position taking 1 in the matrix A. The three matrices are combined together to form a joint matrix with size (3, V, E) input into the network. Among them, the joint matrix is used in channel-specific topology modeling, and three parallel branches model each matrix in the joint matrix, that is, one branch corresponds to one matrix.

[0058] Step three: multi-channel directed spatial convolution module extracts spatial information and aggregates

[0059] This step will input the data from the network into two different branches for spatial dimension information processing. The first branch is the channel-specific topology branch, which first uses dynamic topology to infer the topology structure specific to each channel, then proposes a global attention graph convolution to better model the non-natural connection joints, and extracts the global feature-specific topology structure on the channel through the update function. The second branch is the directed edge convolution branch, which uses average pooling as a time dimension for computational efficiency, then constructs a local graph with local edges based on the K-NN (k-nearest neighbors algorithm) based on the Euclidean distance, and aggregates the local edges and the same edges based on the local edges through trainable parameters. Finally, the information extracted by the two branches is aggregated and sent to the time module.

[0060] The multi-channel directed spatial convolution module is composed of a channel-specific topology branch and a directed edge convolution branch. These two branches are responsible for capturing global channel-specific features and local features, respectively.

[0061] (1) Channel-specific topology modeling. The branch that obtains global features is composed of three parallel sub-branches, which convolve each of the three channels (i.e., node coordinates (x, y, z) three coordinate dimensions). Unlike traditional graph convolution, which directly sums the results, we connect the results in the channel dimension to obtain global channel-specific features.

[0062] We take the three association matrices in the joint matrix as the shared topology of all channels. By taking the structure of the shared topology as prior knowledge, and then refining it through the specific association of each channel, the topology of each channel is modeled. The whole process consists of an extraction function E(·) and an update function U(·). Among them, the extraction function uses global attention graph convolution to model the channel correlation between non-natural connection joints of the input feature X ∈ R T×N×C to obtain the channel correlation matrix Q ∈ R T×NxC . The update function associates the specific channel correlation with the shared topology to obtain the global channel-specific topology S ∈ R T×N×C ,

[0063] S = A k + α · Q (1)

[0064] where A k is the kth channel association matrix, and α is a trainable parameter representing the connection strength.

[0065] Global channel-specific features are obtained by combining the global channel-specific topology S c of each channel with the corresponding channel feature X c , where represents the c-th channel. Each channel responds to the kinematic relationship between joints and bones under different motions.

[0066]

[0067] where || is a concatenation operation. The final output Z varies dynamically with the input.

[0068] (2) Global attention graph convolution. The topological graph A in is multiplied by three learnable weight matrices W Q , W K and W V , to obtain the query Q, key K and value V. The weights of the value V extracted from A in are obtained by calculating the similarity between the query Q and the transpose of the key K, and then the weights and the corresponding value V are weighted and summed to obtain the final attention graph A atten ,

[0069]

[0070] In order to adapt to the specific channel topological structure, we use a multi-head self-attention mechanism, which allows the model to learn relevant information in different representation subspaces,

[0071]

[0072] Global attention uses the neighborhood information as a convolution operation. The update rule is as follows:

[0073]

[0074] where W (l) is the weight matrix of the l-th layer, and σ(·) represents the activation function.

[0075] (3) Directed edge convolution. Directed edge convolution maintains local structure by constructing a local graph, thereby focusing on the aggregation of local features. Therefore, when calculating (updating) the local features of joint v i , the current features of v i , as well as the features of the k joints closest to v i in the current directed topological graph are considered. The k points closest to v i in the feature space can form a local graph, and the edge feature extraction function is used to obtain the edge features of v i and the k nearby nodes. The process is calculated as follows:

[0076]

[0077] ​

[0078] where φ(v i , v k ) represents a linear transformation.

[0079] Step four: Multi-scale contrastive temporal convolution module processes time domain information

[0080] After obtaining the features in the spatial dimension, this module contains 5 branches, which can be divided into four groups. The first group extracts multi-scale temporal features, the second group removes redundant information in the sequence frame, the third group processes the features of the current frame, and the fourth group specially accounts for non-local branches to obtain non-local features. Then the output of the non-local branch is compared with the output of another branch respectively, and the contrastive learning is used to maximize the different features between frames and enhance the feature representation, and enhance the information exchange between the two branches.

[0081] Specifically, the multi-scale contrastive temporal convolution module is used to extract the inter-frame relationship between human behaviors of different durations (i.e. frames). This module contains five branches, which can be divided into four groups. The first group extracts multi-scale temporal features: each branch of this group contains a 1x1 convolution to reduce the channel dimension, and contains a convolution kernel size of 5x1 with different expansion numbers. The second group removes redundant information in the sequence frame: this group contains a 3x1 max pooling layer after the 1x1 convolution to extract the most significant information. The third group processes the features of the current frame: this group only has a 1x1 convolution to process the information of each frame. The fourth group obtains non-local features: a 1x1 adaptive average pooling layer is added at the beginning, and a non-local branch is designed to use the average value to adaptively obtain the representative frame, thereby obtaining a long skip connection in the time dimension.

[0082] Then the InfoNCE loss function is used to calculate the combined loss between the non-local branch and other branches,

[0083]

[0084] where, and are the feature representations of the outputs of the two branches, f D is a discriminator function that scores the consistency between and , defined as the dot product between the two vectors. σ(·) is an activation function. By calculating the combined loss, they can obtain information from each other to improve their performance in feature extraction through convolution operations.

[0085] In addition, since action recognition is a multi-classification task (i.e. each data has a label, but the label has multiple categories), we use CrossEntropyLoss for multi-classification task L mWe unify the multi-classification task and the contrastive learning task into a main- auxiliary learning framework, where the former is the main task and the latter is the auxiliary task. Formally, the joint learning objective is defined as:

[0086] L=L c +βL m (8)

[0087] where β controls the importance of the contrastive learning task.

[0088] Step five: performing classification operation

[0089] This step obtains the result data after ten spatio-temporal convolution block operations, and the format of the data is (NXM, C, T, V), wherein N, M, C, T and V represent the batch size of the data, the number of moving people in the video, the number of channels, the number of frames and the number of nodes respectively. Global average pooling is performed on the data, the average value of all pixel values in each channel graph is obtained, a new channel graph is obtained to achieve the effect of reducing the dimension of the data, and then the Dropout layer is used to inactivate part of the neurons in the network to obtain an output of (output channel number, classification number), and finally the classification is performed through the full connection layer to obtain the score of each class. The scores of the corresponding classes of the three streams are added, and the class with the highest final score is the recognized action.

[0090] Embodiment two

[0091] The purpose of this embodiment is to provide a human behavior recognition system based on multi-channel directed graph convolution.

[0092] A human behavior recognition system based on multi-channel directed graph convolution comprises:

[0093] A data acquisition unit is configured to acquire a human skeleton data sequence to be recognized, and perform corresponding preprocessing to obtain joint stream, skeleton stream and motion stream data.

[0094] A directed graph construction unit is configured to construct a directed graph based on the preprocessed skeleton data to obtain a correlation matrix corresponding to the human skeleton data.

[0095] a feature extraction unit configured to input the joint stream, the skeleton stream and the motion stream into a pre-trained multi-channel directed graph convolution model respectively, wherein the multi-channel directed graph convolution model comprises a plurality of sequentially connected spatio-temporal convolution blocks, each spatio-temporal convolution block comprises a plurality of sequentially connected multi-channel directed spatial convolution modules for extracting spatial information and aggregation and a multi-scale temporal convolution module for obtaining time domain information, and each spatio-temporal convolution block except the first layer further comprises a processing process of merging the feature information obtained by performing residual convolution on the output data of the previous spatio-temporal convolution block and the output of the multi-scale temporal convolution module; in the multi-channel directed spatial convolution module, the correlation matrix is used as a shared topology of all channels, and the topology of each channel is modeled by taking the structure of the shared topology as prior knowledge;

[0096] a classification and recognition unit configured to obtain scores of different categories corresponding to different streams based on the feature information output by the multi-channel directed graph convolution model, and take the category with the maximum score as the final human behavior recognition result by adding the scores of the categories corresponding to different streams.

[0097] Further, the technical details of the system described in the embodiment are described in detail in Embodiment One, and therefore will not be described here.

[0098] In more embodiments, there are also provided:

[0099] An electronic device comprising a memory and a processor, and computer instructions stored in the memory and running on the processor, when the computer instructions are run by the processor, the method described in Embodiment One is completed. For brevity, this will not be described here.

[0100] It should be understood that in the present embodiment, the processor can be a central processing unit CPU, and the processor can also be other general-purpose processors, digital signal processors DSP, application-specific integrated circuits ASIC, ready-to-program gate arrays FPGA or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components, etc. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor, etc.

[0101] The memory can include read-only memory and random access memory, and provide instructions and data to the processor, a part of the memory can also include non-volatile random access memory. For example, the memory can also store device type information.

[0102] A computer readable storage medium for storing computer instructions, when the computer instructions are executed by the processor, the method described in Embodiment One is completed.

[0103] The method in the embodiment one can be directly embodied as being completed by a hardware processor, or being completed by a combination of hardware and software modules in the processor. The software modules can be located in a storage medium in the art, such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory, an electrically erasable programmable memory, a register, or the like. The storage medium is located in a memory, and a processor reads information in the memory, and combines hardware to complete the steps of the above method. To avoid repetition, details are not described here.

[0104] Those skilled in the art can realize that the units and algorithm steps of the examples described in combination with the embodiments can be realized in electronic hardware or a combination of computer software and electronic hardware. Whether the functions are realized in hardware or software mode depends on the specific application and design constraints of the technical solutions. A person skilled in the art can use different methods to realize the described functions for each specific application, but such implementation should not be considered beyond the scope of the present application.

[0105] The human behavior recognition method and system based on multi-channel directed graph convolution provided by the above embodiments can be realized, and has a broad application prospect.

[0106] The above only describes the preferred embodiments of the present application and is not used to limit the present application. For those skilled in the art, the present application can have various modifications and changes. Any modification, equivalent replacement, improvement, etc. within the spirit and principle of the present application shall be included in the protection scope of the present application.

Claims

1. A method for human behavior recognition based on multi-channel directed graph convolution, characterized in that, include: The data sequence of the human skeleton to be identified is obtained and preprocessed accordingly to obtain joint flow, bone flow and motion flow data; A directed graph is constructed based on the preprocessed skeleton data to obtain the correlation matrix corresponding to the human skeleton data; The joint flow, skeletal flow, and motion flow are respectively input into a pre-trained multi-channel directed graph convolutional model. This model includes several sequentially connected spatiotemporal convolutional blocks. Each spatiotemporal convolutional block includes a sequentially connected multi-channel directed spatial convolutional module for extracting and aggregating spatial information, and a multi-scale temporal convolutional module for acquiring temporal information. Additionally, each spatiotemporal convolutional block, except for the first layer, includes a process of merging the feature information from the output data of the previous spatiotemporal convolutional block after residual convolution with the output of the multi-scale temporal convolutional module. In the multi-channel directed spatial convolutional module, the correlation matrix is ​​used as the shared topology for all channels. The structure of the shared topology is used as prior knowledge to model the topology of each channel. The skeleton data is represented as a directed acyclic graph, where joints are vertices and bones are edges. The direction of each edge is determined by the distance between the vertex and the centroid of the skeleton, with vertices closer to the centroid pointing to vertices farther from the centroid. The correlation matrix is ​​chosen to represent the direction of bones between two adjacent joints. Given a given... vertices and For a directed graph with 10 edges, the incidence matrix is ​​a... matrix The incidence matrix consisting of the source vertices is defined as a matrix. The correlation matrix composed of the target vertices is called a matrix. These three matrices, when combined, form a matrix of size [size missing]. The joint matrix is ​​input into the network; By using an incidence matrix to represent the orientation of bones between two adjacent joints, a self-loop is added to the vertex representing the centroid of the skeleton; given a... vertices and A directed graph with edges, then The correlation matrix is ​​a A matrix whose elements represent the corresponding vertices ( ) and edge ( The relationship between ); if yes The source vertex, or yes The passing side, then ;if yes The target vertex, or yes The incoming edge, here If there is no connection, it is 0; to distinguish between source vertices and destination vertices, the incidence matrix composed of source vertices is defined as a matrix. The correlation matrix composed of the target vertices is called a matrix. ; It is a reserved matrix Take the absolute value of the position at -1. It is a reserved matrix The position in the middle that takes a value of 1; Based on the feature information output by the multi-channel directed graph convolution model, a pre-set classifier is used to obtain the scores of each category corresponding to different flows. By adding the scores of the categories corresponding to different flows, the category with the highest score is taken as the final human behavior recognition result.

2. The human behavior recognition method based on multi-channel directed graph convolution as described in claim 1, characterized in that, In each spatiotemporal convolutional block, the input data is processed by a multi-channel directed spatial convolution module to obtain channel-specific global and local features. The global and local features are then fused and input into a multi-scale temporal convolution module to obtain multi-scale temporal features.

3. The human behavior recognition method based on multi-channel directed graph convolution as described in claim 1, characterized in that, The multi-scale temporal convolution module includes a first branch and a second branch arranged in parallel for extracting multi-scale temporal features, a third branch for removing redundant information in sequence frames, a fourth branch for processing the features of the current frame, and a fifth branch for obtaining non-local features.

4. The human behavior recognition method based on multi-channel directed graph convolution as described in claim 3, characterized in that, The output of the fifth branch in the multi-scale temporal convolution module is compared with the outputs of the other branches, and contrastive learning is used to maximize the different features between frames and enhance the feature representation.

5. The human behavior recognition method based on multi-channel directed graph convolution as described in claim 1, characterized in that, The multi-channel directed spatial convolution module includes several channel-specific topology branches and directed edge convolution branches. The channel-specific topology branches first use dynamic topology to infer the specific topology of each channel, then model the joints of non-natural connections through global attention map convolution, and extract the specific topology of global features on the channel through an update function. The directed edge convolution branches aggregate local features by constructing multiple local maps in the skeleton map.

6. The human behavior recognition method based on multi-channel directed graph convolution as described in claim 1, characterized in that, The construction of the directed graph based on the preprocessed skeleton data is as follows: bones are used as edges of the directed graph, joints are used as vertices of the directed graph, and each directed edge points from the vertex closest to the centroid of the skeleton to the vertex furthest from the centroid of the skeleton.

7. The human behavior recognition method based on multi-channel directed graph convolution as described in claim 1, characterized in that, The pre-defined classifier uses a fully connected layer.

8. A human behavior recognition system based on multi-channel directed graph convolution, characterized in that, include: The data acquisition unit is used to acquire the human skeleton data sequence to be identified and perform corresponding preprocessing to obtain joint flow, bone flow and motion flow data. The directed graph construction unit is used to construct a directed graph based on preprocessed skeleton data to obtain the association matrix corresponding to the human skeleton data. A feature extraction unit is used to input the joint flow, skeletal flow, and motion flow into a pre-trained multi-channel directed graph convolutional model. The multi-channel directed graph convolutional model includes several sequentially connected spatiotemporal convolutional blocks. Each spatiotemporal convolutional block includes a sequentially connected multi-channel directed spatial convolutional module for extracting and aggregating spatial information, and a multi-scale temporal convolutional module for acquiring temporal information. Additionally, each spatiotemporal convolutional block, except for the first layer, includes a process of merging the feature information from the output data of the previous spatiotemporal convolutional block after residual convolution with the output of the multi-scale temporal convolutional module. In the multi-channel directed spatial convolutional module, the correlation matrix is ​​used as the shared topology for all channels. The structure of the shared topology is used as prior knowledge to model the topology of each channel. The skeleton data is represented as a directed acyclic graph, where joints are vertices and bones are edges. The direction of each edge is determined by the distance between the vertex and the centroid of the skeleton, with vertices closer to the centroid pointing to vertices farther from the centroid. The correlation matrix is ​​chosen to represent the direction of bones between two adjacent joints. Given a given... vertices and For a directed graph with 10 edges, the incidence matrix is ​​a... matrix The incidence matrix consisting of the source vertices is defined as a matrix. The correlation matrix composed of the target vertices is called a matrix. These three matrices, when combined, form a matrix of size [size missing]. The joint matrix is ​​input into the network; By using an incidence matrix to represent the orientation of bones between two adjacent joints, a self-loop is added to the vertex representing the centroid of the skeleton; given a... vertices and A directed graph with edges, then The correlation matrix is ​​a A matrix whose elements represent the corresponding vertices ( ) and edge ( The relationship between ); if yes The source vertex, or yes The passing side, then ;if yes The target vertex, or yes The incoming edge, here If there is no connection, it is 0; to distinguish between source vertices and destination vertices, the incidence matrix composed of source vertices is defined as a matrix. The correlation matrix composed of the target vertices is called a matrix. ; It is a reserved matrix Take the absolute value of the position at -1. It is a reserved matrix The position in the middle that takes a value of 1; The classification and recognition unit is used to obtain the scores of each category corresponding to different flows based on the feature information output by the multi-channel directed graph convolution model and a pre-set classifier. By adding the scores of the categories corresponding to different flows, the category with the highest score is taken as the final human behavior recognition result.

9. An electronic device, comprising a memory, a processor, and a computer program stored in the memory and running thereon, characterized in that, When the processor executes the program, it implements a human behavior recognition method based on multi-channel directed graph convolution as described in any one of claims 1-7.

10. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When executed by the processor, the program implements a human behavior recognition method based on multi-channel directed graph convolution as described in any one of claims 1-7.

Images