A human pose estimation method based on two-dimensional graph attention network

Abstract

The present invention discloses a method for human posture estimation based on a two-dimensional graph attention network. This method designs a convolution method for a graph neural network and introduces attention between nodes into it, which is named two-dimensional graph attention convolution. This graph convolution can process graph neural networks with two-dimensional node features. It is highly portable and easy to use. It can be connected to any multi-person heatmap generated by bottom-up human posture estimation to improve its accuracy. A reasonable network structure is designed using this convolution to optimize the feature heatmap. This method is not only applicable to the human posture estimation scenario of generating heatmaps, but can also be used to optimize other situations where the node features are two-dimensional nodes and there are different connection relationships between nodes.

Description

Technical Field

[0001] The present invention belongs to the field of computer vision technology, and specifically relates to a human posture estimation method based on a two-dimensional graph attention network. Background Art

[0002] Human pose estimation is a hot and important research topic in the field of computer vision. Its primary objective is to enable computers to locate the key points of each person in an image or video. If there are multiple people in an image, the detected joints must be grouped into different individuals. However, for a long time, the performance of human pose estimation algorithms has struggled to reach practical application levels. However, with the rise of deep neural networks and graph neural networks, the practical applicability of human pose estimation algorithms has dramatically improved. This rapid improvement is largely due to the introduction of deep neural networks and the continuous refinement and optimization of network architectures tailored to the specific human pose estimation task. Graph neural networks are a graph-based deep learning method that leverages the feature extraction capabilities of neural networks to process connected graph-structured data. Graph neural network node features are all one-dimensional, while heatmaps representing key points obtained by deep learning methods for human pose estimation are presented as two-dimensional heatmaps. Forcibly expanding these two-dimensional heatmaps to the one-dimensional representation that traditional graph neural networks can handle destroys the two-dimensional information in the heatmaps. Therefore, there is an urgent need for graph neural networks that can process two-dimensional node features.

[0003] There are two mainstream methods for multi-person human pose estimation: top-down and bottom-up. The top-down method first detects pedestrians in the entire image and then applies a single-person human pose estimation algorithm to estimate key points for each pedestrian. Its disadvantage is that it is time-consuming when there are many people, but its advantage is high accuracy. The bottom-up method first detects all semantic joints in the entire image and then groups the joints belonging to different people. Its disadvantage is slightly lower accuracy, but it is faster. This paper focuses on optimizing the bottom-up human pose estimation method.

[0004] Currently, human pose estimation algorithms using deep learning still face challenges. First, optimizing the heatmaps generated by bottom-up human pose estimation is difficult because the number of joints in each heatmap varies, making network architecture design challenging. Second, networks that achieve high accuracy in human pose estimation often have many parameters. On mobile devices with limited computing power or when real-time estimation is required, network model size reduction is necessary to reduce memory consumption or speed up inference. However, this approach comes at a significant cost in accuracy. Therefore, a method that improves prediction accuracy while introducing fewer parameters is urgently needed. We therefore propose a two-dimensional graph convolution method that can directly process the heatmaps generated by top-down human pose estimation and further design a two-dimensional graph neural network that optimizes these heatmaps. Our algorithm aims to better optimize the heatmaps generated by bottom-up human pose estimation and take into account the connectivity between key points on the human body. Furthermore, our algorithm requires fewer model parameters, improving heatmap prediction accuracy without introducing a large number of parameters.

[0005] Disadvantages of existing technology

[0006] Currently, there's no reliable method for optimizing heatmaps generated by bottom-up multi-person human pose estimation. While fully connected layers can be used to optimize heatmaps, this approach only works for a single person. If the number of joints in a heatmap is uncertain, the number of fully connected layer outputs is also uncertain, making network design difficult. Graph neural networks also fail to optimize heatmaps when the number of joints in a heatmap is uncertain, as their node features have a fixed dimension.

[0007] Graph neural networks process one-dimensional node features in graph structures. They cannot process two-dimensional or higher-dimensional node features. Forcibly converting two-dimensional node features into one-dimensional ones destroys the spatial information of the two-dimensional features. Therefore, graph neural networks are not well suited for graph structures with two-dimensional node features, such as heat maps generated by bottom-up human pose estimation.

[0008] On mobile devices with limited computing power or when real-time estimation is required, network models need to be reduced to reduce memory consumption or speed up inference. However, this behavior comes at the cost of significant accuracy sacrifices. Therefore, a method that can improve prediction accuracy while introducing fewer parameters is urgently needed. Summary of the Invention

[0009] The present invention proposes a human posture estimation method based on a two-dimensional graph attention network, comprising the following steps:

[0010] S1: Construct an adjacency matrix based on the connection relationship of key points of the human body;

[0011] S2: Perform feature transformation on the two-dimensional node features of all joints. The one-dimensional node features are transformed using a learnable two-dimensional matrix, and the two-dimensional node features are transformed using a convolution kernel. Performing feature transformation on all nodes is equivalent to the first step of depthwise separable convolution. The heat map generated by human pose estimation is generated using H (l) ∈R c×h×w Indicates that H i (l) ∈H (l) ,i∈[0,c) where c represents the number of key points, h and w represent the height and width of the heat map, l represents the number of convolution layers, l∈[0,L), L is the total number of convolution layers. (l) Perform feature transformation to obtain the heat map H′ after feature transformation (l) =H (l) *f (l) , H (l) ∈R c ×h′×w′ , f (l) ∈R c×k×k×c Represents the convolution kernel for feature transformation of the l-th layer features, and k represents the size of the convolution kernel;

[0012] S3: Find the attention coefficient between all two-dimensional nodes. Find the attention coefficient between two-dimensional node i and its adjacent node j, and use it with H i ' (l) ,H j ' (l) The feature H′ obtained by splicing in the channel dimension ij (l) =H′ i (l) ||H′ j (l) Convolution kernels of the same size right Perform convolution to get a temporary variable σ represents the leakyRelu activation function, and || represents the concatenation of two feature maps in the channel dimension. Find the relationship between the current node i and all adjacent nodes N i The temporary variable obtained by this operation and normalized is the attention coefficient between the current node and all adjacent nodes. The attention coefficient between node i and node j is

[0013]

[0014] , where the attention coefficient with the own node is multiplied by 10 and then normalized, because the neighboring node affects the own node rather than the neighboring node;

[0015] S4: Node feature update: When node i is updated, all neighboring nodes of node i are first transformed according to step 2, and then the attention coefficients of all nodes are calculated according to step 3. The features after feature transformation are weighted summed according to the attention coefficient, which is the feature value of node i after a two-dimensional graph convolution.

[0016] S5: When updating the feature of node i, the node i updates the multiple attention heads K times with different transformation matrices and different attention coefficients, and takes the average to get the feature update value of a certain section. The number of head parameters increases several times. The feature update value of node i with the multi-attention mechanism is

[0017] is the kth attention mechanism coefficient of the convolution of node i and node j in the lth layer, is the feature value of node i under the k-th attention mechanism in layer l.

[0018] Beneficial effects of the present invention:

[0019] (1) The convolution method of the designed graph neural network can process graph data with two-dimensional node features, which expands the application scope of graph neural networks.

[0020] (2) A 2D residual graph attention network designed with 2D graph attention convolution can optimize and improve the accuracy of heatmaps generated by bottom-up human pose estimation with an uncertain number of joints. This network is not only applicable to human pose estimation scenarios where heatmaps are generated, but can also be used to optimize other scenarios where the node features are 2D nodes with different connection relationships between nodes.

[0021] (3) On mobile devices with insufficient computing power or when real-time estimation is required, it is necessary to reduce the network model to reduce memory consumption or speed up inference, but this behavior comes at the cost of a significant sacrifice in accuracy. Our two-dimensional residual graph attention network introduces only a small number of parameters and can significantly improve accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] Figure 1 Schematic diagram of feature transformation

[0023] Figure 2 Schematic diagram for finding the attention coefficient between nodes

[0024] Figure 3 Update the schematic for the node feature

[0025] Figure 4 Schematic diagram of the two-dimensional residual graph attention network DETAILED DESCRIPTION

[0026] It should be noted that, unless there is a conflict, the embodiments in this application and the features in the embodiments can be combined with each other. The present invention is further described in detail below with reference to the drawings and specific embodiments.

[0027] In order to describe the present invention more clearly, the technical solution of the present invention is described in detail below with reference to the accompanying drawings and specific implementation methods.

[0028] The present invention proposes a human posture estimation method based on a two-dimensional graph attention network, comprising the following steps:

[0029] S1: Construct an adjacency matrix based on the connection relationship of key points of the human body;

[0030] S2: Perform feature transformation on the two-dimensional node features of all joints. The one-dimensional node features are transformed using a learnable two-dimensional matrix, and the two-dimensional node features are transformed using a convolution kernel. Performing feature transformation on all nodes is equivalent to the first step of depthwise separable convolution. The heat map generated by human pose estimation is generated using H (l) ∈R c×h×w Indicates that H i (l) ∈H (l) ,i∈[0,c) where c represents the number of key points, h and w represent the height and width of the heat map, l represents the number of convolution layers, l∈[0,L), L is the total number of convolution layers. (l) Perform feature transformation to obtain the heat map H′ after feature transformation (l) =H (l) *f (l) , H (l) ∈R c ×h′×w′ , f (l) ∈R c×k×k×c Represents the convolution kernel for feature transformation of the l-th layer features, and k represents the size of the convolution kernel, such as Figure 1 As shown;

[0031] S3: Find the attention coefficient between all two-dimensional nodes. Find the attention coefficient between two-dimensional node i and its adjacent node j, using Features obtained by concatenating in the channel dimension Convolution kernels of the same size right Perform convolution to get a temporary variable σ represents the leakyRelu activation function, and || represents the concatenation of two feature maps in the channel dimension. Find the relationship between the current node i and all adjacent nodes N i The temporary variable obtained by this operation and normalized is the attention coefficient between the current node and all adjacent nodes. The attention coefficient between node i and node j is

[0032]

[0033] , where the attention coefficient of the node itself is multiplied by 10 and then normalized, because the neighboring node affects the node itself rather than the node itself, such as Figure 2 As shown;

[0034] S4: Node feature update: When node i is updated, all neighboring nodes of node i are first transformed according to step 2, and then the attention coefficients of all nodes are calculated according to step 3. The features after feature transformation are weighted summed according to the attention coefficient, which is the feature value of node i after a two-dimensional graph convolution. like Figure 3 shown.

[0035] S5: When updating the feature of node i, the node i updates the multiple attention heads K times with different transformation matrices and different attention coefficients, and takes the average to get the feature update value of a certain section. The number of head parameters increases several times. The feature update value of node i with the multi-attention mechanism is is the kth attention mechanism coefficient of the convolution of node i and node j in the lth layer, For the feature value of node i under the k-th attention mechanism in layer l, a reasonable network structure is designed using two-dimensional graph attention convolution to optimize the feature heat map, and the long jump connection of the residual network is added. We call this network a two-dimensional residual graph attention network, as shown in Figure 4 As shown:

[0036] The rectangular box, rounded rectangular box, and cylindrical box in the figure represent the two-dimensional graph attention convolution, BatchNorm normalization layer, and ReLU activation function respectively. The upper connecting line represents the long jump connection, and the plus sign represents the addition of feature maps. N is the number of repetitions. The larger the length and width of the feature heat map, the larger N is.

[0037] The above description of the embodiments is intended to facilitate understanding and application of the present invention by those skilled in the art. It will be apparent that those skilled in the art can readily make various modifications to the above embodiments and apply the general principles described herein to other embodiments without requiring inventive effort. Therefore, the present invention is not limited to the above embodiments, and improvements and modifications made by those skilled in the art based on the disclosure of the present invention should fall within the scope of protection of the present invention.

Claims

1. A human posture estimation method based on two-dimensional graph attention network, characterized by The following steps are included: S1: Construct an adjacency matrix based on the connection relationship of key points of the human body; S2: Perform feature transformation on the two-dimensional node features of all joints, use a learnable two-dimensional matrix to perform feature transformation on the one-dimensional node features, and use convolution to perform feature transformation on the two-dimensional node features. Performing feature transformation on all nodes is equivalent to the first step of depth-wise separable convolution. The heat map generated by human posture estimation is generated by H (l) ∈R c×h×w express, Where c represents the number of key points, h and w represent the height and width of the heat map, l represents the number of convolution layers, l∈[0,L), L is the total number of convolution layers, then for the heat map H (l) Perform feature transformation to obtain the heat map H′ after feature transformation ( l) =H (l) *f (l) , H(l) ∈R c×h′×w′,f (l) ∈R c×k×k×c Represents the convolution kernel for feature transformation of the l-th layer features, and k represents the size of the convolution kernel; S3: Find the attention coefficient between all two-dimensional nodes. When finding the attention coefficient between two-dimensional node i and its adjacent node j, use Features obtained by concatenating in the channel dimension Convolution kernels of the same size right Perform convolution to get a temporary variable σ represents the leakyRelu activation function, || represents the concatenation of two feature maps in the channel dimension, and finds the relationship between the current node i and all adjacent nodes N i The temporary variable obtained by this operation and normalized is the attention coefficient between the current node and all adjacent nodes. The attention coefficient between node i and node j is The attention coefficient of the node itself is multiplied by 10 and then normalized, because the neighboring node affects the node itself rather than influencing it. S4: Node feature update: When node i is updated, all neighboring nodes of node i are first transformed according to step 2, and then the attention coefficients of all nodes are calculated according to step 3. The features after feature transformation are weighted summed according to the attention coefficient, which is the feature value of node i after a two-dimensional graph convolution. S5: When updating the feature, node i updates the multiple attention heads K times with different transformation matrices and different attention coefficients, and takes the average to get the feature update value of a certain section. The number of head parameters will be multiplied several times. The feature update value of node i using the multiple attention mechanism is is the kth attention mechanism coefficient of the convolution of node i and node j in the lth layer, is the feature value of node i under the k-th attention mechanism in layer l.

Images