A Multi-Relation Encoder for 3D Human Pose Estimation

Abstract

The invention discloses a multi-relation encoder for 3D human posture estimation, which relates to the technical fields of computer vision and human-computer interaction; the multi-relation encoder adopts a multi-relation encoder network, which includes an attention module and a graph convolution module; the attention module is improved to an adaptive multi-relation attention module, which includes three matrices, namely, an attention matrix, a joint self-similarity matrix and a predefined bone adjacency matrix, which are respectively used to capture the implicit relationship, explicit relationship and inherent physical topological relationship of the joints, and the adaptive multi-relation attention module is used to capture the global features of the joints; the graph convolution network module is obtained by connecting two GCNs in series, as well as layer normalization and residual connection, and the graph convolution network module is used to capture the local features between adjacent joints. The invention realizes the multi-relation representation learning and global-to-local modeling process of human posture, and obtains accurate three-dimensional posture estimation.

Description

Technical Field

[0001] The present invention belongs to the technical field of computer vision and human-computer interaction, and in particular relates to a multi-relation encoder for 3D human posture estimation. Background Art

[0002] 3D human pose estimation in computer vision is an important and challenging research topic. Its goal is to recover the coordinates of key points of the human body in 3D space from a single image or video frame, which has important application value for many human-computer interaction systems, such as robot control, motion analysis, virtual reality, etc.

[0003] At present, the 3D human posture estimation technology is mainly divided into the structure of direct regression of 3D coordinates and the structure based on 2D posture lifting. The method of direct regression of 3D coordinates is to use convolutional neural network to directly regress 3D coordinates from images, which has the advantage of end-to-end learning, but this type of method has the defect of being unable to model the structural constraint information of the human body. The method based on 2D posture lifting is to first use the existing 2D posture estimation model to predict the 2D key points, and then map them to 3D space, but this type of method is prone to error accumulation.

[0004] In the method based on 2D posture lifting, 2D key points are mapped to 3D space. The 2D joint coordinates detected by the existing 2D key point detector of the human body are used as input. After complex mapping of the deep neural network, the human joint coordinates in 3D space can be finally obtained. For the complex mapping of deep neural networks, the existing technology mainly uses graph convolutional networks and attention mechanisms to model the structural constraint information of the human body, but neither of them can take into account the global and local modeling of human posture at the same time, and there are the following defects.

[0005] Methods based on graph convolutional networks: The introduction of graph convolutional networks can encode the constraint relationship between bones, but there is a problem of weight sharing, which makes it unable to obtain the global dependency between joints. The reasons for the problem are: First, due to the weight sharing restriction, the graph convolutional network is based on the graph structure, and the weights are shared between nodes, which leads to it being able to only extract local neighborhood information and unable to model global dependencies; second, due to insufficient structural prior knowledge, the simple graph convolutional network structure itself lacks prior knowledge of the human body structure, so the constraint and planning capabilities for complex postures are weak; third, due to the limited network expression ability, the existing simple graph convolutional network structure and the number of parameters are limited, and its feature expression ability is relatively weak, making it difficult to accurately model complex postures. In summary, due to the structural restrictions of its shared weights and the limitations of its own structural representation capabilities, the graph convolutional network is unable to learn the global information of human postures, and has weak constraints and fitting capabilities for complex postures. This is also one of the problems that need to be solved in current research.

[0006] Methods based on the attention mechanism: The introduction of the attention mechanism can learn non-local long-range dependencies, but it does not incorporate structural prior knowledge and cannot model various relationships between joints. The reasons for the problem are: First, the lack of structural prior constraints. The attention mechanism itself does not incorporate prior knowledge of human body structure, so it cannot model different types of joint constraint relationships; second, it is difficult for modules to collaborate. Simply stacking different modules cannot achieve good integration and collaboration between modules, so as to give full play to their respective advantages; third, the feature relationship is single. The attention module only models the implicit relationship between joint features, but does not model explicit relationships (similarities and differences of original features) and physical topological relationships. In summary, the attention mechanism itself lacks human body structure priors, and the synergy with other modules is limited, making it unable to effectively model various relationships of human posture, which also limits its effect on the task of three-dimensional posture estimation.

[0007] The present invention proposes a multi-relation encoder for 3D human posture estimation, which realizes multi-relation modeling and global and local representation of human posture. Summary of the invention

[0008] The purpose of the present invention is to provide a multi-relation encoder for 3D human posture estimation, so as to solve the problem that the methods in the prior art proposed in the above background technology cannot simultaneously take into account the global and local modeling of human posture. Among them, the method based on graph convolutional network cannot obtain the global dependency relationship between joints, and the fitting ability of complex postures is weak. The method based on the attention mechanism does not incorporate structural prior knowledge and cannot model multiple relationships between joints. The simple stacking of different modules results in limited effect and cannot give play to the synergistic effect between modules.

[0009] To achieve the above object, the present invention adopts the following technical solutions:

[0010] A multi-relation encoder for 3D human posture estimation, the multi-relation encoder adopts a multi-relation encoder network, the multi-relation encoder network includes an attention module and a graph convolution module;

[0011] The attention module is improved into an adaptive multi-relation attention module, which includes three matrices: an attention matrix, a joint self-similarity matrix, and a predefined bone adjacency matrix. The three matrices are respectively used to capture the implicit relationship, explicit relationship, and inherent physical topological relationship of the joints, and structural prior knowledge is added to the adaptive multi-relation attention module; the adaptive multi-relation attention module is used to capture the global features of the joints;

[0012] The graph convolutional network module is obtained by connecting two GCNs in series, as well as layer normalization and residual connection. The graph convolutional network module is used to capture local features between adjacent joints.

[0013] Preferably, the three matrices are specifically expressed as:

[0014] Given an input X∈R N×C , X is passed into the adaptive multi-relational attention module, where θ(·), φ(·), η(·) are learnable transformations implemented by convolution; the transformation is expressed as:

[0015]

[0016]

[0017]

[0018] in, is the parameterized weight matrix;

[0019] Use a pairwise function f(·) to compute the affinity between all positions, and use the dot product as the operation of f:

[0020] f(θ(X), φ(X)) = θ(X) φ(X) T

[0021] A Attn is the attention matrix used to capture the implicit relationship between joints, A Attn The formula is expressed as:

[0022] A Attn =Softmax(f(θ(X),φ(X)))∈R N×N

[0023] A SSM It is used to capture the explicit relationship between joints and introduces the self-similarity matrix (SSM). SSM The formula is expressed as:

[0024] A SsM =Softmax(f(X,X))∈R N×N

[0025] A Adj represents the predefined bone adjacency matrix, which is used to capture the inherent physical topological relationship between human joints. Adj The formula is expressed as:

[0026] A Adj ∈R N×N .

[0027] Preferably, structural prior knowledge is added to the adaptive multi-relation attention module, specifically:

[0028] A SSMand A Adj As prior knowledge, with A Attn Combined to obtain the spatial dependent attention matrix (SpDA), denoted as A SpDA ; Use the Concat[·,·] operation to connect A Attn , A SSM and A Adj ; The formula of SpDA is:

[0029] A SpDA =Softmax(δ(Concat[A SSM A Attn , A Adj ]))

[0030] where δ(·) represents a learnable transformation implemented via convolution; thus, the non-local contextual response Y∈R N×C It is calculated by two linear mappings as follows:

[0031]

[0032] Here, ρ(·) and η(·) are learnable transformations implemented via convolution.

[0033] Preferably, the design of GCN in the graph convolutional network module is to classify adjacent nodes according to their semantic information, then perform feature transformation and aggregate the transformed features; the GCN formula is expressed as follows:

[0034]

[0035] Where K is the index of the adjacent node type, W K is the filter matrix of the Kth type of 1-hop neighboring nodes, A K is the K-th type of adjacency matrix, D K is the degree matrix of the Kth type.

[0036] Preferably, the multi-relation encoder network further includes a loss function module, and the loss function is a mean square error, expressed as:

[0037]

[0038] Where N represents the number of joints, i is the index of the joint type, and They are represented as the ground truth 3D joint position and the predicted 3D joint position respectively.

[0039] The method for estimating a 3D human posture using the multi-relation encoder comprises the following steps:

[0040] S1. Input a single image, use the 2D posture model to predict the 2D key points, and generate the 2D joint point coordinates P from the single image. 2D ∈R N×2 ;

[0041] S2. Construct a multi-relation encoder network, including an adaptive multi-relation attention module, a graph convolutional network module, and a loss function module;

[0042] S3, in the adaptive multi-relation attention module, three types of joint relations are learned simultaneously, namely implicit relations, explicit relations and inherent physical topological relations;

[0043] S4, effectively integrate the adaptive multi-relation attention module and the graph convolution module to form a global-local structure;

[0044] S5. Set the 2D joint point coordinates P 2D ∈R N×2 As input, where N is the number of joint points, a multi-relation encoder network is used to extract the features of the joint points, and the 3D joint point coordinates P are predicted based on the learned global and local features of the joint points. 3D ∈R N×3 ; Finally, perform three-dimensional coordinate regression and output three-dimensional posture.

[0045] Compared with the prior art, the present invention has the following beneficial effects:

[0046] (1) The multi-relation encoder for 3D human posture estimation in the present invention adopts a multi-relation encoder network. The attention module learns different relations, and the graph convolution module provides local features. The two realize global and local collaborative modeling. The multi-relation encoder network integrates the attention module with the graph convolution module to form a collaborative network of global representation and local constraints. Through this global-to-local process, multi-relation modeling of human posture is realized; finally, the network integrates multiple information to perform 3D posture prediction;

[0047] (2) The attention module of the present invention designs an adaptive multi-relation attention structure, and simultaneously models the implicit relationship, explicit relationship and physical topological structure relationship of joint features;

[0048] (3) In the present invention, structural prior knowledge is added to the attention module: the self-similarity matrix between joints and the predefined bone adjacency matrix to learn different types of relationships between joints. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] Figure 1 A system block diagram of a multi-relation encoder for 3D human posture estimation in the present invention;

[0050] Figure 2A structural block diagram of an adaptive multi-relation attention module in a multi-relation encoder for 3D human pose estimation in the present invention;

[0051] Figure 3 This is a visualization comparison result diagram of the 3D human posture estimation method using the multi-relation encoder in the present invention and the most advanced method HTNet on the Human3.6M dataset in Example 1. DETAILED DESCRIPTION

[0052] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0053] Embodiment 1:

[0054] See also Figure 1 , Multi-relation Encoder for 3D Human Pose Estimation,The Multi-relation Encoder (MuRE) captures the multi-level dependencies between human joints through a global-to-local architecture. MuRE consists of an adaptive multi-relation attention module (AMuRA) and a graph convolutional network module (GCN Block).

[0055] In this embodiment, the adaptive multi-relation attention module is mainly composed of three matrices: attention matrix, predefined bone adjacency matrix and joint self-similarity matrix. These matrices are used to capture the implicit relationship, explicit relationship and inherent physical topological relationship of joints respectively. The details are as follows:

[0056] Scaled dot product attention: The input of the self-attention mechanism is the query Q, key K and value V, where N j and N d Represent the number of joints and channels respectively. The traditional self-attention mechanism is expressed as:

[0057]

[0058] where d is the dimension of Q, K and V, Used for normalization.

[0059] like Figure 2As shown in the figure, the adaptive multi-relation attention module (AMuRA) improves on the traditional attention mechanism. Specifically, the predefined skeleton adjacency matrix is ​​integrated into the attention mechanism to capture the inherent physical topological relationship between joints. In addition, the joint self-similarity matrix (SSM) is introduced to capture the explicit spatial relationship between joints. Finally, the attention matrix generated by query and key captures the implicit relationship between joints. The aggregation of these three matrices enables AMuRA to adaptively capture various relationships between human joints, thereby achieving a higher level of global representation.

[0060] The adaptive multi-relation attention (AMuRA) in the present invention can adaptively capture three kinds of relationships between human joints: implicit relationships, explicit relationships and inherent physical topological relationships. These three relationships can be represented by the following three matrices:

[0061] Given an input X∈R N×C , pass X into AMuRA, where θ(·), φ(·), It is a learnable transformation achieved through convolution. The transformation can be expressed as:

[0062]

[0063]

[0064]

[0065] in, is a parameterized weight matrix. Then a pairwise function f(·) is used to calculate the affinity between all positions, and the dot product is used as the operation of f:

[0066] f(θ(X), φ(X)) = θ(X) φ(X) T

[0067] A Attn It is an attention matrix used to capture the implicit relationship between joints, and its formula can be expressed as:

[0068] A Attn =Softmax(f(θ(X),φ(X)))∈R N×N

[0069] In order to capture the explicit relationship between joints, a self-similarity matrix (SSM) is introduced, using A SSM It means that the formula can be expressed as:

[0070] A SSM =Softmax(f(X,X))∈R N×N

[0071] Use AAdj Represents a predefined adjacency matrix that captures the inherent physical topological relationships between human joints. Adj The formula is expressed as:

[0072] A Adj ∈R N×N

[0073] A SSM and A Adj As a priori knowledge, compare them with A Attn Combined to obtain the spatial dependent attention matrix (SpDA), denoted as A SpDA . Use the Concat[·,·] operation to concatenate A Attn , A SSM and A Adj The formula of SpDA is:

[0074] A SpDA =Softmax(δ(Concat[A SSM , A Attn , A Adj ]))

[0075] where δ(·) represents a learnable transformation implemented by convolution. Therefore, the non-local contextual response Y∈R N×C It can be calculated by two linear mappings as follows:

[0076]

[0077] Among them, ρ(·) and It is a learnable transformation achieved through convolution.

[0078] In this embodiment, the graph convolutional network module is specifically as follows:

[0079] Graph Convolutional Network Module: Given an undirected graph G = {V, E}, where V represents a node set and E∈{0, 1} N×N It is the adjacency matrix that records the edges between nodes, and the number of nodes is represented by N. Ordinary GCN can be expressed as:

[0080]

[0081] Among them, H∈R N×C and H∈R N×C′ Respectively represent the input and output feature matrices of GCN, W∈R C×C′ represents the learnable weight matrix, C and C′ represent the number of input channels and output channels, respectively. In addition, represents the adjacency matrix of the graph after adding self-connection, where I is the identity matrix, is a diagonal matrix.

[0082] The graph convolutional network module in the present invention is obtained by connecting two GCNs in series, as well as layer normalization and residual connection. The design of the GCN used in the GCN Block is to classify the adjacent nodes according to their semantic information, and then perform feature transformation and aggregate the transformed features. The present invention only considers the spatial semantic information of 1-hop adjacent nodes. The GCN formula is expressed as follows:

[0083]

[0084] Where K is the index of the adjacent node type, W K is the filter matrix of the Kth type of 1-hop neighboring nodes, A K is the K-th type of adjacency matrix, D K is the degree matrix of the Kth type.

[0085] In this embodiment, the loss function module is as follows:

[0086] For the model, the loss function is the mean squared error:

[0087]

[0088] Where N represents the number of joints and i is the index of the joint type. The ground truth 3D joint position and the predicted 3D joint position are represented as and

[0089] When the present invention performs a 3D human posture estimation method, the multi-relation encoder (MuRE) uses a ready-made 2D posture model to generate 2D joint point coordinates P from a single image. 2D ∈R N×2 As input, N is the number of joints. Then, MuRE is used to extract the features of the joints. Finally, based on the learned global and local features of the joints, the 3D joint coordinates P are predicted. 3D ∈R N×3 .

[0090] Adaptive multi-relational attention (AMuMA) is designed in the multi-relational encoder (MuRE) of the present invention. Specifically, a predefined skeletal adjacency matrix is ​​added to the attention mechanism to capture the inherent physical topological relationship between joints; a joint self-similarity matrix (SSM) is added to capture the spatial explicit relationship between joints; finally, the attention matrix generated by Query and Key captures the implicit relationship between joints. The aggregation of the three matrices enables AMuRA to adaptively capture multiple relationships between human joints, thereby obtaining a higher global representation capability. At the same time, the present invention introduces GCN Block to capture local features between adjacent joints, and combines it with AMuRA to obtain a multi-relational encoder (MuRE). MuRE realizes global to local multi-relational modeling of human joints. The present invention realizes the multi-relational representation learning and global to local modeling process of human posture by integrating attention and graph convolutional network, thereby obtaining accurate three-dimensional posture estimation.

[0091] Experimental verification:

[0092] The present invention uses PyTorch to implement the above-mentioned multi-relation encoder (MuRE) of the present invention, and trains for 30 epochs with a batch size of 1024 on a single NVIDIAGeForce RTX 2080 SUPER GPU. The model depth N is set to 3, the initial learning rate is set to 0.0005, and 0.95 and 0.5 are used as decay factors after each epoch and every 5 epochs, respectively, and finally the Adam optimizer is used.

[0093] We evaluate the model on two 3D human pose estimation datasets: Human3.6M and MPI-INF-3DHP.

[0094] Human3.6M is the most widely used indoor dataset for 3D human pose estimation, containing 3.6 million images taken by four cameras with different viewpoints. It includes 15 different actions and 11 subjects. The proposed model uses the mean position error per joint (MPJPE) as the evaluation metric.

[0095] MPI-INF-3DHP adds data enhancement for foreground and background in the dataset, including three different scenes: green background (GS), non-green background (noGS), and outdoor. The generalization ability of MuRE is evaluated by testing on MPI-INF-3DHP, using the correctness of key point detection (PCK) and area under the curve (AUC) as indicators.

[0096] We compare MuRE with the previous state-of-the-art methods on the Human3.6M dataset. As shown in Table 1, our method is quantitatively compared with the state-of-the-art methods on Human3.6M under MPJPE using 2D poses detected by Cascaded Pyramid Networks (CPN). Indicates that a refining method was used.

[0097] As shown in Table 1, the method in this invention achieves a performance of 48.5mm on MPJPE. In addition, MGCN uses a refinement module to further improve performance, and the method in this invention also adds this module, and MuRE reaches 47.6mm on MPJPE, which is a significant improvement over MGCN (relative improvement of 1.8%).

[0098] Table 1 Quantitative comparison of the method in this invention with other methods on the Human3.6M dataset

[0099]

[0100]

[0101] In addition, as shown in Table 2, quantitative comparisons are performed on Human3.6M under MPJPE using the true (GT) 2D joint locations as input.

[0102] As shown in Table 2, with ground-truth (GT) 2D joint points as input, MuRE achieves the best performance of 33.2mm, which is a relative improvement of 0.8% compared with the state-of-the-art methods.

[0103] Table 2 Quantitative comparison of methods using ground-truth (GT) 2D joint positions as input

[0104]

[0105] In the cross-dataset evaluation scenario, the present invention is only trained on the Human3.6M training dataset and tested on the MPI-INF-3dHP test dataset, comparing MuRE with the previous state-of-the-art methods. As shown in Table 3, the present invention gives the respective results of different backgrounds, GS means green screen, noGS means non-green screen, and Outdoor means outdoor.

[0106] As shown in Table 3, the model in the present invention achieved the best results in both PCK and AUC indicators.

[0107] Table 3 Quantitative comparison of MPI-INF-3DHP

[0108]

[0109]

[0110] The experimental results in Table 1, Table 2 and Table 3 show that the present invention is reasonable and effective, and MuRE achieves the most advanced performance. Figure 3 The visualization results of MuRE on the Human3.6M dataset are also shown. Compared with the state-of-the-art HTNet method, MuRE can predict more reasonable and accurate 3D poses. This shows that the fusion of the adaptive multi-relation attention module (AMuRA) and the graph convolutional network module significantly improves the modeling ability of complex poses.

[0111] The above description is only used to help understand the method of the present invention and its core essence, but the protection scope of the present invention is not limited thereto. For those skilled in the art in the art, equivalent replacement or change according to the technical solution and inventive concept of the present invention within the technical scope disclosed by the present invention should be included in the protection scope of the present invention. In summary, the content of this specification should not be understood as limiting the present invention.

Claims

1. A multi-relation encoder for 3D human posture estimation, wherein the multi-relation encoder uses a multi-relation encoder network, wherein the multi-relation encoder network includes an attention module and a graph convolutional network module; characterized in that: The attention module is improved into an adaptive multi-relation attention module, which includes three matrices: an attention matrix, a joint self-similarity matrix, and a predefined bone adjacency matrix. The three matrices are respectively used to capture the implicit relationship, explicit relationship, and inherent physical topological relationship of the joints, and structural prior knowledge is added to the adaptive multi-relation attention module; the adaptive multi-relation attention module is used to capture the global features of the joints; The graph convolutional network module is obtained by connecting two GCNs in series, and performing layer normalization and residual connection. The graph convolutional network module is used to capture local features between adjacent joints. The 3D human body posture estimation method comprises the following steps: S1. Input a single image, use the 2D posture model to predict the 2D key points, and generate 2D joint point coordinates from a single image ; S2. Construct a multi-relation encoder network, including an adaptive multi-relation attention module, a graph convolutional network module, and a loss function module; S3, in the adaptive multi-relation attention module, three types of joint relations are learned simultaneously, namely implicit relations, explicit relations and inherent physical topological relations; S4, effectively integrate the adaptive multi-relation attention module and the graph convolution module to form a global-local structure; S5. 2D joint point coordinates As input, where N is the number of joints, a multi-relation encoder network is used to extract the features of the joints, and the 3D joint coordinates are predicted based on the learned global and local features of the joints. ; Finally, perform three-dimensional coordinate regression and output three-dimensional posture.

2. A multi-relation encoder for 3D human pose estimation according to claim 1, characterized in that: The three matrices are specifically expressed as: Given an input , pass X into the adaptive multi-relation attention module, where , , is a learnable transformation achieved through convolution; the transformation is expressed as: in, , , is the parameterized weight matrix; Using the pairwise function f to compute the affinity between all positions and use the dot product as the operation for f: is the attention matrix used to capture the implicit relationship between joints, The formula is expressed as: It is used to capture the explicit relationship between joints and introduces the self-similar matrix SSM. The formula is expressed as: represents a predefined bone adjacency matrix that captures the inherent physical topological relationships between human joints. The formula is expressed as: 。 3. A multi-relation encoder for 3D human pose estimation according to claim 2, characterized in that: Add structural prior knowledge to the adaptive multi-relation attention module, specifically: Will and As prior knowledge, Combined to obtain the spatial dependent attention matrix SpDA, expressed as ; Use the Concat[·,·] operation to connect , and ; The formula of SpDA is: in, represents a learnable transformation implemented by convolution; therefore, the non-local context response Y It is calculated by two linear mappings as follows: in, and It is a learnable transformation achieved through convolution.

4. The multi-relation encoder for 3D human pose estimation according to claim 1, characterized in that: The design of GCN in the graph convolutional network module is to classify adjacent nodes according to their semantic information, then perform feature transformation and aggregate the transformed features; the GCN formula is expressed as follows: Where K is the index of the adjacent node type, is the filter matrix of the K-th type of 1-hop neighboring nodes, is the K-th type of adjacency matrix, is the degree matrix of the Kth type.

5. The multi-relation encoder for 3D human pose estimation according to claim 1, characterized in that: The multi-relation encoder network also includes a loss function module, and the loss function is the mean square error, which is expressed as: Where N represents the number of joints, i is the index of the joint type, and They are represented as the ground truth 3D joint position and the predicted 3D joint position respectively.

Images