Human behavior recognition method and system based on heterogeneous skeleton graph

Abstract

The application belongs to the technical field of human behavior recognition, and provides a human behavior recognition method and system based on a heterogeneous skeleton graph, which comprises the following steps: initializing skeleton data as a heterogeneous skeleton graph; in the heterogeneous skeleton graph, a vertex represents a joint, an edge represents an actual link and a virtual link between joints, the actual link is initialized as a bone naturally connected between two joints, the virtual link is initialized as full connection between skeleton root tree levels, a vertex set and an actual link edge set form an actual meta path, and a vertex set and a virtual link edge set form a virtual meta path; a joint stream, a bone stream and a motion stream are input into a self-attention heterogeneous graph convolution network together with the heterogeneous skeleton graph to recognize a behavior category; the self-attention heterogeneous graph convolution network adopts three sub-networks to process the joint stream, the bone stream and the motion stream respectively, and each sub-network extracts features from the actual meta path and the virtual meta path respectively. This is favorable to improving the accuracy of human behavior recognition.

Description

Technical Field

[0001] This invention belongs to the field of human behavior recognition technology, and particularly relates to a human behavior recognition method and system based on heterogeneous skeleton graphs. Background Technology

[0002] The statements in this section are merely background information related to the present invention and do not necessarily constitute prior art.

[0003] Human behavior recognition refers to the task of identifying specific actions of a person in a video by using specific algorithms, based on the characteristics of the human body itself and the features of the actions performed. The basic process of behavior recognition generally includes: preprocessing of image data, detection of moving human bodies, extraction of motion features, training and classification of features, and behavior recognition.

[0004] However, traditional recognition methods are based on RGB datasets, which are susceptible to environmental factors such as background and lighting. Human skeleton joint data, on the other hand, represents high-level human features, virtually unaffected by appearance, and better avoids noise from background, lighting, and viewing angle changes, exhibiting excellent robustness. Furthermore, it is highly efficient in terms of computation and storage. Therefore, research on human behavior recognition methods based on skeleton data is not only significant in theoretical academic research but also has considerable value in practical applications.

[0005] Human behavior recognition algorithms based on skeleton data typically input a set of coordinates representing the positions of skeleton joints. Common deep learning methods for processing skeleton joints are mainly divided into three types: Convolutional Neural Networks (CNNs), Recurrent Neural Networks (RNNs), and Graph Convolutional Networks (GCNs). Currently, methods based on CNNs and RNNs have been widely developed. However, the skeleton is an irregular graph topology, which makes it difficult for CNNs and RNNs, which are more suited to regular data, to extract skeleton information. Therefore, Graph Convolutional Networks, developed specifically for graph topologies, have become the mainstream approach.

[0006] However, current graph-based convolutional network methods use isomorphic graphs, treating physical and non-physical connections between joints as edges of the same attribute and modeling them simultaneously. This is inconsistent with reality. The connection between two joints includes two types: actual connection and virtual connection. Actual connection refers to the natural connection between bones in the human skeleton; virtual connection refers to the interaction relationship that exists in a specific action. For example, the relationship between the two hands in the action of "clapping," or the relationship between the hand and head in the action of "wearing glasses." The impact of actual and virtual connections is different in different actions. Isomorphic graphs model these two different types of connections simultaneously, blurring the boundaries between them. Furthermore, the virtual connection relationship between two nodes is mostly acquired autonomously by the model through attention or other mechanisms, with fewer constraints. This increases the uncertainty in the training process, making the model prone to convergence to local optima, which is detrimental to the accuracy of human behavior recognition. Summary of the Invention

[0007] To address the technical problems mentioned above, this invention provides a human behavior recognition method and system based on heterogeneous skeleton graphs. Compared to traditional graph convolution methods, a heterogeneous skeleton graph with nodes having the same attributes but edges with different attributes is designed, which helps to improve the accuracy of human behavior recognition.

[0008] To achieve the above objectives, the present invention adopts the following technical solution:

[0009] The first aspect of the present invention provides a method for human behavior recognition based on heterogeneous skeleton graphs.

[0010] Human behavior recognition methods based on heterogeneous skeleton graphs include:

[0011] Obtain skeleton data;

[0012] The skeleton data is preprocessed to obtain joint flow, bone flow, and motion flow;

[0013] The skeleton data is initialized as a heterogeneous skeleton graph; in the heterogeneous skeleton graph, vertices represent joints, and edges represent actual and virtual links between joints. Actual links are initialized as bones that are naturally connected between two joints, and virtual links are initialized as full connections between the levels of the rooted tree of the skeleton. The vertex set and the actual link edge set form actual meta-paths, and the vertex set and the virtual link edge set form virtual meta-paths.

[0014] The joint flow, skeletal flow, and motion flow are combined with the heterogeneous skeleton. Figure 1 The input is a self-attention heterogeneous graph convolutional network, which identifies the behavior category to which the skeleton data belongs. The self-attention heterogeneous graph convolutional network uses three sub-networks to process joint flow, skeleton flow and motion flow respectively, and each sub-network extracts features from the actual meta-path and virtual meta-path respectively.

[0015] Furthermore, the self-attention heterogeneous graph convolutional network obtains the query, key, and value by multiplying the actual meta-path or virtual meta-path with three learning matrices, obtains the weights by calculating the similarity between the transposes of the query and the key, obtains the self-attention graph by weighting the weights and values, and uses the self-attention graph as neighborhood information for graph convolution to obtain global features.

[0016] Furthermore, the self-attention heterogeneous graph convolutional network performs global average pooling on the global features to generate local and non-local features. Based on the local and non-local features, it generates importance weights and then performs feature aggregation on each channel of the actual meta-path or virtual meta-path through the importance weights to obtain the final features.

[0017] Furthermore, the self-attention heterogeneous graph convolutional network fuses the features of the actual meta-path and the virtual meta-path, and then uses multi-scale temporal convolutional blocks to extract multi-scale temporal information.

[0018] Furthermore, the virtual link is initialized as follows:

[0019]

[0020]

[0021] in, Indicates a virtual link, L represents the number of levels in the rooted tree of the skeleton, H... n This represents the nth level of a rooted tree with a skeleton. Represented as H n To H n+1 A fully connected edge, where ∥ represents a join operation.

[0022] Furthermore, the joint flow includes: batch number, channel number, frame number, node number, and number of moving people in a frame.

[0023] Furthermore, the skeletal flow is the coordinate difference between different joints; or, the motion flow is the coordinate difference of the same joint between adjacent frames.

[0024] A second aspect of the present invention provides a human behavior recognition system based on heterogeneous skeleton graphs.

[0025] A human behavior recognition system based on heterogeneous skeleton graphs includes:

[0026] The data acquisition module is configured to acquire skeleton data.

[0027] The preprocessing module is configured to preprocess the skeleton data to obtain joint flow, bone flow, and motion flow.

[0028] The graph construction module is configured to initialize the skeleton data into a heterogeneous skeleton graph. In the heterogeneous skeleton graph, vertices represent joints, and edges represent actual and virtual links between joints. Actual links are initialized as bones that are naturally connected between two joints, and virtual links are initialized as full connections between the rooted tree levels of the skeleton. The vertex set and the actual link edge set form actual meta-paths, and the vertex set and the virtual link edge set form virtual meta-paths.

[0029] The identification module is configured to: compare the joint flow, skeletal flow, and motion flow with the heterogeneous skeleton. Figure 1 The input is a self-attention heterogeneous graph convolutional network, which identifies the behavior category to which the skeleton data belongs. The self-attention heterogeneous graph convolutional network uses three sub-networks to process joint flow, skeleton flow and motion flow respectively, and each sub-network extracts features from the actual meta-path and virtual meta-path respectively.

[0030] A third aspect of the present invention provides a computer-readable storage medium.

[0031] A computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the human behavior recognition method based on heterogeneous skeleton graphs as described in the first aspect above.

[0032] A fourth aspect of the present invention provides a computer device.

[0033] A computer device includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor, when executing the program, implements the steps of the human behavior recognition method based on heterogeneous skeleton graphs as described in the first aspect above.

[0034] Compared with the prior art, the beneficial effects of the present invention are:

[0035] Compared with traditional graph convolution methods, this invention designs a heterogeneous skeleton graph with nodes having the same attributes but edges with different attributes, which is beneficial to improving the accuracy of human behavior recognition.

[0036] This invention designs a self-attention heterogeneous graph convolutional network for heterogeneous skeleton graphs, which learns the features of actual and virtual meta-paths respectively, and uses a semantic aggregation module to achieve their optimal combination to improve the accuracy of human behavior recognition. Attached Figure Description

[0037] The accompanying drawings, which form part of this invention, are used to provide a further understanding of the invention. The illustrative embodiments of the invention and their descriptions are used to explain the invention and do not constitute an improper limitation of the invention.

[0038] Figure 1This is a structural diagram of the heterogeneous skeleton diagram shown in Embodiment 1 of the present invention;

[0039] Figure 2 This is a schematic diagram of an embodiment of the present invention, showing that the actual edge is initialized as a bone naturally connected between two joints;

[0040] Figure 3 This is a schematic diagram illustrating how virtual edges are initialized as complete connections between adjacent layers of a skeleton root tree, as shown in Embodiment 1 of the present invention.

[0041] Figure 4 This is the overall architecture diagram of the self-attention heterogeneous graph convolutional network shown in Embodiment 1 of the present invention;

[0042] Figure 5 This is a schematic diagram illustrating the specific operations of the self-attention heterogeneous graph convolutional block for different meta-paths, as shown in Embodiment 1 of the present invention.

[0043] Figure 6 This is a schematic diagram of the inter-frame adaptive convolution module shown in Embodiment 1 of the present invention;

[0044] Figure 7 This is a structural diagram of the multi-scale temporal convolution module shown in Embodiment 1 of the present invention. Detailed Implementation

[0045] The present invention will be further described below with reference to the accompanying drawings and embodiments.

[0046] It should be noted that the following detailed description is illustrative and intended to provide further explanation of the invention. Unless otherwise specified, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains.

[0047] It should be noted that the terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the scope of exemplary embodiments according to the invention. As used herein, the singular form is intended to include the plural form as well, unless the context clearly indicates otherwise. Furthermore, it should be understood that when the terms "comprising" and / or "including" are used in this specification, they indicate the presence of features, steps, operations, devices, components, and / or combinations thereof.

[0048] It should be noted that the flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of methods and systems according to various embodiments of the present invention. It should be noted that each block in a flowchart or block diagram may represent a module, segment, or portion of code, which may include one or more executable instructions for implementing the logical functions specified in the various embodiments. It should also be noted that in some alternative implementations, the functions marked in the blocks may occur in a different order than that shown in the drawings. For example, two consecutively represented blocks may actually be executed substantially in parallel, or they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the flowcharts and / or block diagrams, and combinations of blocks in the flowcharts and / or block diagrams, may be implemented using a dedicated hardware-based system that performs the specified functions or operations, or using a combination of dedicated hardware and computer instructions.

[0049] Example 1

[0050] This embodiment provides a method for human behavior recognition based on heterogeneous skeleton graphs.

[0051] This embodiment provides a human behavior recognition method based on heterogeneous skeleton graphs. Compared to traditional graph convolution methods, it designs a heterogeneous skeleton graph with nodes having the same attributes but edges with different attributes. Vertices represent joints, and edges represent actual and virtual links (i.e., actual edges and virtual edges) between joints. Actual edges are initialized as bones naturally connecting two joints. Virtual edges are initialized as fully connected layers within the rooted tree of the skeleton. The initialized virtual links are equivalent to regularizing the model based on prior human knowledge, which helps the model converge to the global optimum. Nodes and the two types of edges respectively form two types of meta-paths: actual meta-paths and virtual meta-paths.

[0052] The behavior recognition method based on self-attention heterogeneous graph convolutional networks comprises three information flow sub-networks: joint flow, bone flow, and motion flow. Each sub-network stacks nine basic self-attention heterogeneous graph convolutional blocks. Each self-attention heterogeneous graph convolutional block consists of a self-attention adaptive spatial convolutional module, a semantic aggregation module, a multi-scale temporal convolutional module, and a residual connection. Notably, the first self-attention heterogeneous graph convolutional block of each sub-network does not have a residual connection.

[0053] Specifically, the self-attention adaptive spatial convolution module includes self-attention meta-path convolution, three parallel branches, and inter-frame adaptive convolution. For a given meta-path, the self-attention meta-path convolution enables nodes to learn the attention values ​​of their neighbors based on the meta-path, acquiring the intrinsic topology of the meta-path and using this topology as the neighborhood node information for the convolution. The inter-frame adaptive convolution adaptively adjusts the weights of frames in the meta-path, providing temporal modeling capabilities for spatial convolution. For each frame, importance weights are adaptively generated based on local and global features in the temporal dimension, increasing attention to regions with significant changes in meta-path features. Between meta-paths, a semantic aggregation module is proposed to understand the importance of each meta-path for different actions and assign appropriate weights. This obtains the optimal combination between meta-paths to minimize co-occurring features. The fused features are then input into a multi-scale temporal convolution module to extract rich temporal information to obtain multi-scale temporal features. After training through layers of networks, global average pooling is used to prevent overfitting, and finally, a fully connected layer is used for classification.

[0054] The human behavior recognition method based on heterogeneous skeleton graphs provided in this embodiment includes the following steps:

[0055] Step 1: Obtain skeleton data and preprocess the skeleton data to obtain joint flow, bone flow and motion flow.

[0056] The dataset used in this embodiment is the NTU-RGB+D 60 / 120 dataset, which consists of multiple text files. Each text file contains the number of frames of skeleton data, the number of people performing the action, and the (x,y,z) coordinates of each joint.

[0057] The preprocessing of the skeleton data is to extract specific information such as bone point coordinates, frame length, and number of joints required for network training. Finally, the Dataset and Dataloader modules provided by PyTorch are used to encapsulate it into a format that can be input into the self-attention heterogeneous graph convolutional network, which is a five-dimensional vector with dimensions (N, C, T, V, M). Here, N represents the number of batches in one training session, C represents the number of channels, T represents the number of frames, V represents the number of nodes, and M represents the number of moving people in a frame. This is the input data for the joint flow.

[0058] Taking 3D skeletal data as an example, given two joints v1 = (x1, y1, z1) and v2 = (x2, y2, z2), the skeletal data is represented as the difference between the two joint vectors, that is, the input data of the skeletal flow is e. 1,2 = (x1-x2, y1-y2, z1-z2). For the same joint, the coordinates of adjacent frames are (x1-x2, y1-y2, z1-z2). t ,y t ,z t ) and (xt+1 ,y t+1 ,z t+1 For a given joint, the motion data of the joint can be represented by the coordinate difference between different frames of the same joint, that is, the input data of the motion flow is m. t =(x t+1 -x t ,y t+1 -y t ,z t+1 -z t Then, the input data for skeletal flow and motion flow are calculated using the above formulas and input into the corresponding sub-networks.

[0059] Step 2: Construction of heterogeneous skeleton graph, that is, initializing the skeleton data into a heterogeneous skeleton graph.

[0060] To reduce uncertainty during training, virtual links between joints need to be initialized. Taking into account various possible connection relationships between distant joints and the redundancy of fully connected joints, the skeleton is converted into a rooted skeleton tree, and the fully connected nodes between levels are initialized as virtual links.

[0061] like Figure 1 As shown, the skeleton data is represented as a heterogeneous skeleton graph, where vertices represent joints and edges represent actual and virtual links (i.e., actual and virtual edges) between joints; as... Figure 2 As shown, the actual edges are initialized as bones naturally connected between two joints; as Figure 3 As shown, virtual edges are initialized as fully connected layers between the rooted tree skeleton. These initialized virtual links are equivalent to regularizing the model based on prior human knowledge, which helps the model converge to the global optimum. Nodes and the two types of edges form two types of metapaths: actual metapaths and virtual metapaths.

[0062] For a given skeleton, when initializing virtual edges, the first step is to determine the center of the skeleton, usually the centroid of the skeleton. Following the hierarchical structure of the skeleton, each unvisited neighbor joint connected to the center is visited sequentially. Then, starting from each of these neighbor joints, their neighbor joints are visited sequentially, ensuring that the neighbor joints of the first visited joint are visited before the neighbor joints of the last visited joint are visited. This process is repeated until all neighbor joints of the visited joints in the skeleton have been visited. The tree structure generated in this process is the rooted tree of the skeleton.

[0063] Assuming the skeleton has a rooted tree with L levels, then virtual links... It can be initialized as:

[0064]

[0065]

[0066] Among them, Set n Represented as an edge set, H n This represents the nth level of a rooted tree with a skeleton. Represented as H n+1 To H n+2 A fully connected edge, where ∥ represents a join operation.

[0067] Heterogeneous skeleton diagram is represented as Having a vertex set Edge set ε and edge type set Vertex set Represents all joints. Edge Indicates joint and joints Between attributes The edge, of which This indicates that the edge is an actual edge. It still belongs to virtual edge N represents the number of joints.

[0068] Vertex set And the actual edge set (actual linked edge set) This forms the actual metapath φ A Vertex set And virtual edge sets (virtual linked edge sets) Virtual metapath φ is formed v Adjacency matrix The neighbors of a meta-path can be better represented mathematically, and their elements Metapath Inferior joint and The correspondence. If This indicates a joint. Metapath Inferior joint The neighbor. If This indicates no connection.

[0069] Step 3: Integrate joint flow, bone flow, and motion flow with a heterogeneous skeleton. Figure 1The input is a self-attention heterogeneous graph convolutional network, which identifies the behavior category to which the skeleton data belongs. The self-attention heterogeneous graph convolutional network consists of three sub-networks: joint flow and heterogeneous skeleton graph, input to the first sub-network; skeletal flow and heterogeneous skeleton graph, input to the second sub-network; and motion flow and heterogeneous skeleton graph, input to the third sub-network. The scores of each action obtained from each sub-network are weighted and summed to obtain the behavior category to which the skeleton data belongs. Each sub-network includes several spatiotemporal convolutional blocks (each spatiotemporal convolutional block includes a self-attention adaptive spatial convolutional module, a semantic aggregation module, and a multi-scale temporal convolutional module connected in sequence), global average pooling, Dropout layers, and fully connected layers.

[0070] Step 301: Self-attention adaptive spatial convolution module extracts spatial information.

[0071] The self-attention adaptive spatial convolution module was designed to extract the intrinsic topology of the metapath, obtain channel-specific global features, and emphasize frames with significant variations so that the spatial convolution can incorporate temporal dynamic features.

[0072] The data input to the network is first processed through self-attention meta-path convolutions to learn global features by considering the combined influence of joints on neighboring nodes in both actual and virtual meta-paths. These global features are then fed into three parallel branches, each performing convolutions on different channels to extract channel-specific features. Furthermore, the results of each branch are not simply added together but concatenated along the channel dimension. Simultaneously, the global features are adaptively weighted across frames in inter-frame adaptive convolutions, emphasizing frames with significant changes, enabling spatial convolutions to effectively capture temporal dynamics. Finally, the extracted information is aggregated and fed into a semantic aggregation module.

[0073] (1) Self-attention meta-path convolution.

[0074] like Figure 5 As shown, the attention metapath convolution uses a self-attention mechanism, inferring the internal topology by learning the attention values ​​of neighbors, which is called a self-attention graph. The self-attention graph can represent the connection strength of joints based on metapaths, and can better capture complex structures and rich semantic information.

[0075] metapath With three learning matrices W Q W K and W V Multiply to obtain query Q, key K, and value V; obtain weights by calculating the similarity between the transposes of Q and K; obtain the final self-attention graph by weighting the weights and corresponding values ​​V.

[0076]

[0077] Then, this information is used as neighborhood information for graph convolution, and the update rules are as follows:

[0078]

[0079] Where X and Y are the input feature matrices (joint flow, skeletal flow, or motion flow) and the output feature matrix (i.e., global features), I is the identity matrix, and D is... The degree matrix is ​​W, which is the learnable weight matrix, and σ(·) represents the activation function.

[0080] (2) Inter-frame adaptive convolution.

[0081] like Figure 6 As shown, global average pooling is performed in the spatial, spatial, and temporal dimensions to generate local features Z. local ∈R C×T×1 Nonlocal features Z non-local ∈R C×1×1 To model the context information:

[0082]

[0083]

[0084] To enhance the inter-frame domain, non-local features are incorporated into the calibration weight generation process to further integrate temporal context information. The base weights W are initialized using pre-trained weights. base ∈R C×1×1 Using calibration weight W cali ∈R C′×C×1×1 Perform matrix multiplication to generate importance weights W. import ∈R C′×C×T :

[0085]

[0086] Where Ψ(·) represents the calibration weight generation function, and FC(·) represents the linear mapping function. This represents the element-wise product of broadcasts.

[0087] (3) Channel aggregation.

[0088] Feature aggregation is performed on each channel of the meta-path, and the final feature is obtained by concatenating the output features of all channels. In summary, the output of the entire self-attention adaptive spatial convolution module is computed as follows:

[0089]

[0090] Where ∥ represents a join operation. Let θ represent the metapath of the (c)th channel, and let Θ(·) represent the convolution operation.

[0091] Step 302: The semantic aggregation module aggregates features extracted from different meta-paths.

[0092] Heterogeneous skeleton graphs involve diverse and complex semantic information, typically reflected through different meta-paths. Different semantic features can be extracted from these different meta-paths. Treating features extracted from different meta-paths equally is impractical, as it leads to co-occurring features and diminishes the proportion of useful information. Therefore, a semantic aggregation module is proposed to understand the importance of each meta-path for different actions and assign appropriate weights to them, thereby obtaining the optimal combination of meta-paths.

[0093] To obtain the optimal combination of meta-paths, the semantic aggregation module first transforms the dimensions of the features extracted from a given meta-path, introducing a trainable semantic-level attention vector to obtain the importance of the meta-path to a given node. This operation is then applied to all nodes, and the average value is calculated to obtain the importance of the meta-path to all nodes. The weight of a given meta-path is calculated using softmax based on its importance. The weights of different meta-paths are then weighted and summed with their corresponding features to aggregate the features extracted from different meta-paths.

[0094] For metapath First, the node feature dimensions are transformed using a nonlinear transformation. This results in learnable semantic-level vectors. It is used to measure the importance of node features, as a similarity after dimensional transformation. Therefore, metapath For joints The importance can be calculated as:

[0095]

[0096] in, Represents a node The features output after the self-attention adaptive spatial convolution module are q, a learnable semantic level vector, W, a learnable weight matrix, and b, a learnable bias vector. They are shared by all joints in all metapaths.

[0097] Metapath The importance is obtained by averaging the importance of all nodes in the metapath and normalizing it using the softmax function:

[0098]

[0099] Use what you've learned As weights, features from different meta-paths can be fused to obtain the final feature Z, as shown below:

[0100]

[0101] in, Indicates the type of metapath.

[0102] Step 303: Extract multi-scale temporal information from multi-scale temporal convolutional blocks.

[0103] After obtaining the spatial features, this module contains four branches. The first two branches extract multi-scale temporal features, the third branch removes redundant information from the sequence frames, and the fourth branch processes the features of the current frame.

[0104] like Figure 7 As shown, the multi-scale temporal convolution module is used to extract inter-frame relationships between actions of different durations (i.e., frames). This module contains four branches. The first and second branches extract multi-scale temporal features: each branch contains a 1×1 convolution to reduce channel dimensions and a convolution with a kernel size of 5×1 and different dilation numbers. The third branch removes redundant information from the sequence of frames: this branch contains a 3×1 max-pooling layer after the 1×1 convolution to extract the most salient information. The fourth branch processes the features of the current frame: this group uses only a single 1×1 convolution to process the information from each frame.

[0105] Step 304: Perform the classification operation.

[0106] like Figure 4 As shown, the resulting data, after being processed through nine spatiotemporal convolutional blocks (each including a sequentially connected self-attention adaptive spatial convolutional module, semantic aggregation module, and multi-scale temporal convolutional module), is in the format (N×M,C,T,V), where N, M, C, T, and V represent the batch size, number of moving people in the video, number of channels, number of frames, and number of nodes, respectively. Global average pooling is performed on this data, averaging all pixel values ​​in each channel map to obtain a new channel map, achieving dimensionality reduction. A Dropout layer then deactivates some neurons in the network, resulting in an output of size (number of output channels, number of categories). Finally, a fully connected layer is used for classification, obtaining a score for each category. The scores of corresponding categories in the three sub-networks are weighted and summed; the category with the highest final score is the recognized action.

[0107] Y = W joint ·Y joint +W bone ·Y bone +W motion ·Y motion

[0108] Among them, Wjoint W bone and W motion This represents the learnable weights of the three subnetworks.

[0109] Example 2

[0110] This embodiment provides a human behavior recognition system based on heterogeneous skeleton graphs.

[0111] A human behavior recognition system based on heterogeneous skeleton graphs includes:

[0112] The data acquisition module is configured to acquire skeleton data.

[0113] The preprocessing module is configured to preprocess the skeleton data to obtain joint flow, bone flow, and motion flow.

[0114] The graph construction module is configured to initialize the skeleton data into a heterogeneous skeleton graph. In the heterogeneous skeleton graph, vertices represent joints, and edges represent actual and virtual links between joints. Actual links are initialized as bones that are naturally connected between two joints, and virtual links are initialized as full connections between the rooted tree levels of the skeleton. The vertex set and the actual link edge set form actual meta-paths, and the vertex set and the virtual link edge set form virtual meta-paths.

[0115] The identification module is configured to: compare the joint flow, skeletal flow, and motion flow with the heterogeneous skeleton. Figure 1 The input is a self-attention heterogeneous graph convolutional network, which identifies the behavior category to which the skeleton data belongs. The self-attention heterogeneous graph convolutional network uses three sub-networks to process joint flow, skeleton flow and motion flow respectively, and each sub-network extracts features from the actual meta-path and virtual meta-path respectively.

[0116] It should be noted that the above modules implement the same examples and application scenarios as the steps in Embodiment 1, but are not limited to the content disclosed in Embodiment 1. It should also be noted that the above modules, as part of a system, can be executed in a computer system such as a set of computer-executable instructions.

[0117] Example 3

[0118] This embodiment provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps in the human behavior recognition method based on heterogeneous skeleton graphs as described in Embodiment 1 above.

[0119] Example 4

[0120] This embodiment provides a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, it implements the steps in the human behavior recognition method based on heterogeneous skeleton graphs as described in Embodiment 1 above.

[0121] The above description is merely a preferred embodiment of the present invention and is not intended to limit the invention. Various modifications and variations can be made to the present invention by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A human behavior recognition method based on heterogeneous skeleton graphs, characterized in that, include: Obtain skeleton data; The skeleton data is preprocessed to obtain joint flow, bone flow, and motion flow; Initialize the skeleton data as a heterogeneous skeleton graph; In the heterogeneous skeleton graph, vertices represent joints, and edges represent actual and virtual links between joints. Actual links are initialized as bones that are naturally connected between two joints, and virtual links are initialized as fully connected between the levels of the rooted tree of the skeleton. The vertex set and the actual link edge set form actual meta-paths, and the vertex set and the virtual link edge set form virtual meta-paths. The joint flow, skeletal flow, and motion flow, together with the heterogeneous skeleton graph, are input into a self-attention heterogeneous graph convolutional network to identify the behavior category to which the skeleton data belongs. The self-attention heterogeneous graph convolutional network uses three sub-networks to process the joint flow, skeletal flow, and motion flow respectively, and each sub-network extracts features from the actual meta-path and the virtual meta-path respectively. The self-attention heterogeneous graph convolutional network obtains the query, key, and value by multiplying the actual meta-path or virtual meta-path with three learning matrices. It then obtains weights by calculating the similarity between the transposes of the query and key, and weights and values ​​to obtain a self-attention map. This self-attention map is used as neighborhood information for graph convolution to obtain global features. The network performs global average pooling on these global features to generate local and non-local features. Based on these local and non-local features, it generates importance weights and then performs feature aggregation on each channel of the actual meta-path or virtual meta-path using these importance weights to obtain the final features. The virtual link is initialized as follows: ； ； in, This represents a virtual link, where L represents the number of levels in the rooted tree of the skeleton. This represents the nth level of a rooted tree with a skeleton. express arrive Fully connected edges, This indicates a join operation.

2. The human behavior recognition method based on heterogeneous skeleton graphs according to claim 1, characterized in that, The self-attention heterogeneous graph convolutional network fuses the features of the actual meta-path and the virtual meta-path, and then uses multi-scale temporal convolutional blocks to extract multi-scale temporal information.

3. The human behavior recognition method based on heterogeneous skeleton graphs according to claim 1, characterized in that, The joint flow includes: batch number, channel number, frame number, node number, and number of moving people in a frame.

4. The human behavior recognition method based on heterogeneous skeleton graphs according to claim 1, characterized in that, The skeletal flow is the coordinate difference between different joints; or, the motion flow is the coordinate difference of the same joint between adjacent frames.

5. A human behavior recognition system based on heterogeneous skeleton graphs, characterized in that, include: The data acquisition module is configured to acquire skeleton data. The preprocessing module is configured to preprocess the skeleton data to obtain joint flow, bone flow, and motion flow. The graph construction module is configured to initialize the skeleton data into a heterogeneous skeleton graph. In the heterogeneous skeleton graph, vertices represent joints, and edges represent actual and virtual links between joints. Actual links are initialized as bones that are naturally connected between two joints, and virtual links are initialized as full connections between the rooted tree levels of the skeleton. The vertex set and the actual link edge set form actual meta-paths, and the vertex set and the virtual link edge set form virtual meta-paths. The recognition module is configured to input the joint flow, skeletal flow, and motion flow, together with the heterogeneous skeleton graph, into a self-attention heterogeneous graph convolutional network to identify the behavior category to which the skeleton data belongs; the self-attention heterogeneous graph convolutional network uses three sub-networks to process the joint flow, skeletal flow, and motion flow respectively, and each sub-network extracts features from the actual meta-path and the virtual meta-path respectively. The self-attention heterogeneous graph convolutional network obtains the query, key, and value by multiplying the actual meta-path or virtual meta-path with three learning matrices. It then obtains weights by calculating the similarity between the transposes of the query and key, and weights and values ​​to obtain a self-attention map. This self-attention map is used as neighborhood information for graph convolution to obtain global features. The network performs global average pooling on these global features to generate local and non-local features. Based on these local and non-local features, it generates importance weights and then performs feature aggregation on each channel of the actual meta-path or virtual meta-path using these importance weights to obtain the final features. The virtual link is initialized as follows: ； ； in, This represents a virtual link, where L represents the number of levels in the rooted tree of the skeleton. This represents the nth level of a rooted tree with a skeleton. express arrive Fully connected edges, This indicates a join operation.

6. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it implements the steps in the human behavior recognition method based on heterogeneous skeleton graphs as described in any one of claims 1-4.

7. A computer device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the steps in the human behavior recognition method based on heterogeneous skeleton graphs as described in any one of claims 1-4.

Images