A method, device and system for estimating 3D posture of human body

Abstract

The embodiment of the present invention discloses a method, device and system for estimating 3D posture of a human body. Based on the Transformer encoder architecture and considering the relative distance of 3D key points of a human body, 3D posture estimation of a human body is realized. A single picture is input and the coordinates of the 3D key points of the person in the picture are output. The accuracy of 3D posture estimation is high.

Description

Technical Field

[0001] The embodiments of the present invention relate to the field of artificial intelligence technology, and specifically to a method, device and system for estimating 3D posture of a human body. Background Art

[0002] Human 3D posture estimation is a technology that recognizes the three-dimensional movements of a human body from a two-dimensional image. It has important application value and practical significance for human movement discrimination, intention recognition, behavior detection, sports teaching, etc. The present invention proposes a human 3D posture estimation method based on the Transformer encoder architecture and considering the relative distance of the 3D key points of the human body. It inputs a single image and outputs the coordinates of the 3D key points of the person in the image. Summary of the invention

[0003] To this end, the embodiments of the present invention provide a method, device and system for estimating 3D posture of a human body, which realizes 3D posture estimation of a human body based on the Transformer encoder architecture and takes into account the relative distance of 3D key points of a human body, inputs a single image, and outputs the coordinates of the 3D key points of the person in the image.

[0004] In order to achieve the above objectives, the embodiments of the present invention provide the following technical solutions:

[0005] According to a first aspect of an embodiment of the present invention, a method for estimating a 3D posture of a human body is proposed, the method comprising:

[0006] Detect k 2D key point coordinates representing human body postures from the image to be recognized, where k is the predefined number of human body key points;

[0007] Input k 2D key point coordinates into a two-layer fully connected neural network, extract the 256-dimensional embedding feature vector corresponding to each key point coordinate, and output a total of k 256-dimensional embedding feature vectors;

[0008] The k 256-dimensional embedding feature vectors are input into the transformer encoder. The transformer encoder has a total of 4 encoder_layer layers. Each encoder_layer layer outputs k 256-dimensional vectors, and the output k 256-dimensional feature vectors are input into the next encoder_layer layer in sequence. The k 256-dimensional vectors output by the last encoder_layer layer are respectively input into two layers of fully connected neural networks. The output of the network is the 3D coordinates of each key point.

[0009] Furthermore, the calculation process of each encoder_layer layer includes the following steps:

[0010] a. Input k 256-dimensional feature vectors into the multi-head attention layer to obtain k 256-dimensional feature vectors;

[0011] b. Then add the k 256-dimensional vectors obtained in step a to the k 256-dimensional feature vectors input by the encoder_layer layer to obtain k 256-dimensional feature vectors;

[0012] c. Then perform LayerNorm operation on the k 256-dimensional feature vectors obtained in step b to obtain normalized k 256-dimensional vectors;

[0013] d. Then, the k 256-dimensional vectors obtained in step c are input into a two-layer fully connected neural network to obtain k 256-dimensional vectors, where the k vectors share a fully connected feedforward network;

[0014] e. Finally, add the k vectors obtained in step c and step d respectively, and perform LayerNorm operation on the added k vectors to obtain k normalized 256-dimensional vectors, and input these k 256-dimensional feature vectors into the next encoder_layer layer.

[0015] Furthermore, k 256-dimensional feature vectors are input into the multi-head attention layer to obtain k 256-dimensional feature vectors, including:

[0016] Each 256-dimensional feature vector is transformed into three 256-dimensional feature vectors Q through three transformation matrices i ,K i ,V i , i∈(1,2,3……k), we get a total of 3 tensors Q,K,V, where the shapes of Q,K,V are all (k,256);

[0017] Transform Q, K, V into tensors of shape (k, 8, 32), and further permute them into tensors PQ, PK, PV of shape (8, k, 32), where 8 is the number of heads in the multi-head attention layer and 32 is the dimension of the feature vector of each head;

[0018] Decompose PQ into 8 matrices PQ of shape (k,32) i , i∈(1,2,3…8), decompose PK into 8 matrices PK of shape (k,32) i , i∈(1,2,3…8), 8 matrices PQ of shape (k,32) i Respectively with 8 matrices PK of shape (k,32) i Transpose and multiply to get 8 matrices of shape (k,k), which are divided by Get 8 attention coefficient matrices attn_1 of shape (k, k);

[0019] The relative distance between two human joints is defined as the number of bones connecting the two joints on the skeleton diagram, and this number of bones constitutes the shortest distance between the two joints. The relative distance matrix of a human skeleton diagram consisting of k joints is denoted by R, with a shape of (k, k), and its (i, j)th element represents the relative distance from the i-th joint to the j-th joint.

[0020] Each relative distance value corresponds to an 8-dimensional weight vector, where 8 is the number of heads. Different relative distances correspond to different weight vectors. Each relative distance in R is used to query the corresponding weight vector to obtain a tensor of shape (k, k, 8), which is further permuted into a tensor of shape (8, k, k), denoted as RW.

[0021] Decompose RW into 8 matrices of shape (k,k) divided by Get the attention coefficient matrix attn_2;

[0022] The obtained 8 attention coefficient matrices attn_1 and attn_2 are added to obtain 8 matrices of shape (k, k). The columns of the matrix are further softmaxed to obtain the final 8 attention coefficient matrices ATTN i , i∈(1,2,3…8);

[0023] Decompose the obtained PV into 8 matrices PVi (i from 1 to 8) of shape (k, 32), and decompose 8 matrices ATTN of shape (k, k) i Multiply them with the transpose of 8 matrices of shape (k, 32) respectively to obtain 8 matrices of shape (k, 32). These 8 matrices of shape (k, 32) are transformed into a tensor of shape (8, k, 32), which is further permuted into a tensor of shape (k, 8, 32), and further transformed into a tensor of shape (k, 256).

[0024] The obtained tensor is decomposed into k 256-dimensional feature vectors, and a transformation matrix is ​​input to obtain k 256-dimensional feature vectors.

[0025] Furthermore, k 2D key point coordinates representing human body postures are detected for the image to be recognized, where k is the number of predefined human body key points, specifically including:

[0026] By inputting the image to be identified into the MaskRCNN network, the coordinates of k 2D key points representing the human body posture are obtained.

[0027] Furthermore, the method further comprises:

[0028] If there are multiple people in the image to be identified, the image will be cropped into a single person image before processing.

[0029] According to a second aspect of an embodiment of the present invention, a human 3D posture estimation device is provided, wherein the device comprises:

[0030] 2D human posture detection module, which is used to input k 2D key point coordinates into a two-layer fully connected neural network, extract the 256-dimensional embedding feature vector corresponding to each key point coordinate, and output a total of k 256-dimensional embedding feature vectors;

[0031] The 3D human posture estimation module is used to input the k 256-dimensional embedding feature vectors into the transformer encoder. The transformer encoder has a total of 4 encoder_layer layers. Each encoder_layer layer outputs k 256-dimensional vectors, and the output k 256-dimensional feature vectors are sequentially input into the next encoder_layer layer. The k 256-dimensional vectors output by the last encoder_layer layer are respectively input into two layers of fully connected neural networks. The output of the network is the 3D coordinates of each key point.

[0032] According to a third aspect of an embodiment of the present invention, a human 3D posture estimation system is provided, the system comprising: a processor and a memory;

[0033] The memory is used to store one or more program instructions;

[0034] The processor is used to run one or more program instructions to execute any of the methods described above.

[0035] According to a fourth aspect of an embodiment of the present invention, a computer storage medium is proposed, wherein the computer storage medium contains one or more program instructions, and the one or more program instructions are used to be executed by a human 3D posture estimation system as described in any of the above methods.

[0036] The embodiments of the present invention have the following advantages:

[0037] A method, device and system for estimating 3D human posture proposed in the embodiments of the present invention are based on the Transformer encoder architecture and take into account the relative distance of 3D key points of the human body to achieve 3D human posture estimation. A single image is input and the coordinates of the 3D key points of the person in the image are output. The accuracy of 3D posture estimation is high. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] In order to more clearly illustrate the implementation methods of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for the implementation methods or the description of the prior art. Obviously, the drawings in the following description are only exemplary, and for ordinary technicians in this field, other implementation drawings can be derived from the provided drawings without creative work.

[0039] Figure 1 A schematic diagram of a flow chart of a method for estimating 3D posture of a human body provided in Example 1 of the present invention;

[0040] Figure 2 A human joint skeleton diagram in a human 3D posture estimation method provided in Example 1 of the present invention. DETAILED DESCRIPTION

[0041] The following is a description of the implementation of the present invention by specific embodiments. People familiar with the art can easily understand other advantages and effects of the present invention from the contents disclosed in this specification. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0042] Example 1

[0043] like Figure 1 As shown, this embodiment proposes a method for estimating a 3D posture of a human body, the method comprising:

[0044] S100 , detecting k 2D key point coordinates representing human body postures from the image to be recognized, where k is a predefined number of human body key points.

[0045] In this embodiment, the image to be identified is input into the MaskRCNN network to obtain k 2D key point coordinates representing the human posture, which are recorded as p1 = (x1, y1), p2 = (x2, y2), ... pk = (xk, yk). MaskRCNN can also be replaced by other 2D human posture detectors.

[0046] S200, input k 2D key point coordinates into a two-layer fully connected neural network, extract the 256-dimensional embedding feature vector corresponding to each key point coordinate, and output a total of k 256-dimensional embedding feature vectors. The input of the fully connected neural network N has k nodes, and the input of the i-th node (i from 1 to k) is as follows: (p1, p2..., pk).

[0047] S300, input k 256-dimensional embedding feature vectors into the transformer encoder, the transformer encoder has 4 encoder_layer layers, each encoder_layer layer outputs k 256-dimensional vectors, and the output k 256-dimensional feature vectors are sequentially input into the next encoder_layer layer, and the k 256-dimensional vectors output by the last encoder_layer layer are respectively input into two layers of fully connected neural networks, the output of which is the 3D coordinate of each key point. Among them, each vector shares a fully connected network (that is, k vectors share the weight of a fully connected neural network).

[0048] Furthermore, the calculation process of each encoder_layer layer includes the following steps:

[0049] a. Input k 256-dimensional feature vectors into the multi-head attention layer to obtain k 256-dimensional feature vectors;

[0050] b. Then add the k 256-dimensional vectors obtained in step a to the k 256-dimensional feature vectors input by the encoder_layer layer to obtain k 256-dimensional feature vectors;

[0051] c. Then perform LayerNorm operation on the k 256-dimensional feature vectors obtained in step b to obtain normalized k 256-dimensional vectors;

[0052] d. Then, the k 256-dimensional vectors obtained in step c are input into a two-layer fully connected neural network to obtain k 256-dimensional vectors, where the k vectors share a fully connected feedforward network;

[0053] e. Finally, add the k vectors obtained in step c and step d respectively, and perform LayerNorm operation on the added k vectors to obtain k normalized 256-dimensional vectors, and input these k 256-dimensional feature vectors into the next encoder_layer layer.

[0054] Furthermore, k 256-dimensional feature vectors are input into the multi-head attention layer to obtain k 256-dimensional feature vectors, including:

[0055] Each 256-dimensional feature vector is transformed into three 256-dimensional feature vectors Q through three transformation matrices i ,K i ,V i , i∈(1,2,3……k), we get a total of 3 tensors Q,K,V, where the shapes of Q,K,V are all (k,256);

[0056] Reshape Q, K, V into tensors of shape (k, 8, 32), and further permute them into tensors PQ, PK, PV of shape (8, k, 32), where 8 is the number of heads in the multi-head attention layer and 32 is the dimension of the feature vector of each head;

[0057] Decompose PQ into 8 matrices PQ of shape (k,32) i , i∈(1,2,3…8), decompose PK into 8 matrices PK of shape (k,32) i , i∈(1,2,3…8), 8 matrices PQ of shape (k,32) i Respectively with 8 matrices PK of shape (k,32) i Transpose and multiply to get 8 matrices of shape (k,k), which are divided by Get 8 attention coefficient matrices attn_1 of shape (k, k);

[0058] The relative distance between two human joints is defined as the number of bones connecting the two joints on the skeleton diagram, and this number of bones constitutes the shortest distance between the two joints. According to this definition, Figure 2 The skeleton graph composed of 17 joint points (k=17) shown in the figure has a relative distance matrix denoted by R, with a shape of (k,k), and its (i,j)th element represents the relative distance from the i-th joint point to the j-th joint point;

[0059] The following is Figure 2 The relative distance matrix of the shown bone structure:

[0060] [[0,1,2,3,1,2,3,1,2,3,4,3,4,5,3,4,5],

[0061] [1,0,1,2,2,3,4,2,3,4,5,4,5,6,4,5,6],

[0062] [2,1,0,1,3,4,5,3,4,5,6,5,6,7,5,6,7],

[0063] [3,2,1,0,4,5,6,4,5,6,7,6,7,8,6,7,8],

[0064] [1,2,3,4,0,1,2,2,3,4,5,4,5,6,4,5,6],

[0065] [2,3,4,5,1,0,1,3,4,5,6,5,6,7,5,6,7],

[0066] [3,4,5,6,2,1,0,4,5,6,7,6,7,8,6,7,8],

[0067] [1,2,3,4,2,3,4,0,1,2,3,2,3,4,2,3,4],

[0068] [2,3,4,5,3,4,5,1,0,1,2,1,2,3,1,2,3],

[0069] [3,4,5,6,4,5,6,2,1,0,1,2,3,4,2,3,4],

[0070] [4,5,6,7,5,6,7,3,2,1,0,3,4,5,3,4,5],

[0071] [3,4,5,6,4,5,6,2,1,2,3,0,1,2,2,3,4],

[0072] [4,5,6,7,5,6,7,3,2,3,4,1,0,1,3,4,5],

[0073] [5,6,7,8,6,7,8,4,3,4,5,2,1,0,4,5,6],

[0074] [3,4,5,6,4,5,6,2,1,2,3,2,3,4,0,1,2],

[0075] [4,5,6,7,5,6,7,3,2,3,4,3,4,5,1,0,1],

[0076] [5,6,7,8,6,7,8,4,3,4,5,4,5,6,2,1,0]]

[0077] Each relative distance value corresponds to an 8-dimensional weight vector (a learnable parameter in training), where 8 is the number of heads. Different relative distances correspond to different weight vectors. Each relative distance in R is used to query the corresponding weight vector to obtain a tensor of shape (k, k, 8), which is further permuted into a tensor of shape (8, k, k), denoted as RW.

[0078] Decompose RW into 8 matrices of shape (k,k) divided by Get the attention coefficient matrix attn_2;

[0079] The obtained 8 attention coefficient matrices attn_1 and attn_2 are added to obtain 8 matrices of shape (k, k). The columns of the matrix are further softmaxed to obtain the final 8 attention coefficient matrices ATTN i , i∈(1,2,3…8);

[0080] Decompose the obtained PV into 8 matrices PVi (i from 1 to 8) of shape (k, 32), and decompose 8 matrices ATTN of shape (k, k) i Multiply the 8 (k, 32)-shaped matrices by transpose to obtain 8 (k, 32)-shaped matrices. These 8 (k, 32)-shaped matrices are transformed into a (8, k, 32)-shaped tensor. The 8 (k, 32)-shaped matrices are further permuted into a (k, 8, 32)-shaped tensor, which is further reshaped into a (k, 256)-shaped tensor.

[0081] The obtained tensor is decomposed into k 256-dimensional feature vectors, and a transformation matrix is ​​input to obtain k 256-dimensional feature vectors.

[0082] Furthermore, the method also includes: if there are multiple people in the picture to be identified, the picture is cropped into a single-person image before processing.

[0083] A method for estimating a 3D human body posture is proposed in this embodiment. It is based on the Transformer encoder architecture and takes into account the relative distance of 3D key points of the human body to achieve 3D human body posture estimation. It inputs a single image and outputs the coordinates of the 3D key points of the person in the image. The accuracy of 3D posture estimation is high.

[0084] Example 2

[0085] Corresponding to the above-mentioned embodiment 1, this embodiment proposes a human body 3D posture estimation device, characterized in that the device includes:

[0086] 2D human posture detection module, which is used to input k 2D key point coordinates into a two-layer fully connected neural network, extract the 256-dimensional embedding feature vector corresponding to each key point coordinate, and output a total of k 256-dimensional embedding feature vectors;

[0087] The 3D human posture estimation module is used to input k 256-dimensional embedding feature vectors into the transformer encoder. The transformer encoder has a total of 4 encoder_layer layers. Each encoder_layer layer outputs k 256-dimensional vectors, and the output k 256-dimensional feature vectors are input to the next encoder_layer layer in turn. The k 256-dimensional vectors output by the last encoder_layer layer are input into two layers of fully connected neural networks. The output of the network is the 3D coordinates of each key point.

[0088] The functions performed by the various components in the human body 3D posture estimation device provided by the embodiment of the present invention have been described in detail in the above embodiment 1, so they will not be described in detail here.

[0089] Example 3

[0090] Corresponding to the above embodiment, this embodiment proposes a human 3D posture estimation system, the system comprising: a processor and a memory;

[0091] The memory is used to store one or more program instructions;

[0092] The processor is used to run one or more program instructions to execute the method of embodiment 1 above.

[0093] Example 4

[0094] Corresponding to the above embodiment, this embodiment proposes a computer storage medium, which contains one or more program instructions, and the one or more program instructions are used to be executed by a human 3D posture estimation system as the method of embodiment 1.

[0095] Although the present invention has been described in detail above by general description and specific embodiments, it is obvious to those skilled in the art that some modifications or improvements can be made to the present invention. Therefore, these modifications or improvements made without departing from the spirit of the present invention all belong to the scope of protection claimed by the present invention.

Claims

1. A method for estimating 3D posture of a human body, characterized in that: The method comprises: Detect k 2D key point coordinates representing human body postures from the image to be recognized, where k is the predefined number of human body key points; Input k 2D key point coordinates into a two-layer fully connected neural network, extract the 256-dimensional embedding feature vector corresponding to each key point coordinate, and output a total of k 256-dimensional embedding feature vectors; The k 256-dimensional embedding feature vectors are input into the transformer encoder. The transformer encoder has 4 encoder_layer layers. Each encoder_layer layer outputs k 256-dimensional vectors. The k 256-dimensional feature vectors output are input into the next encoder_layer layer in sequence. The k 256-dimensional vectors output by the last encoder_layer layer are input into two layers of fully connected neural networks. The output of the network is the 3D coordinates of each key point. The calculation process of each encoder_layer layer includes the following steps: a. Input k 256-dimensional feature vectors into the multi-head attention layer to obtain k 256-dimensional feature vectors; b. Then add the k 256-dimensional vectors obtained in step a to the k 256-dimensional feature vectors input by the encoder_layer layer to obtain k 256-dimensional feature vectors; c. Then perform LayerNorm operation on the k 256-dimensional feature vectors obtained in step b to obtain normalized k 256-dimensional vectors; d. Then, the k 256-dimensional vectors obtained in step c are input into a two-layer fully connected neural network to obtain k 256-dimensional vectors, where the k vectors share a fully connected feedforward network; e. Finally, add the k vectors obtained in step c and step d respectively, and perform LayerNorm operation on the added k vectors to obtain k normalized 256-dimensional vectors, and input these k 256-dimensional feature vectors into the next encoder_layer layer.

2. A method for estimating 3D posture of a human body according to claim 1, characterized in that: Input k 256-dimensional feature vectors into the multi-head attention layer to obtain k 256-dimensional feature vectors, including: Each 256-dimensional feature vector is transformed into three 256-dimensional feature vectors Q through three transformation matrices i ,K i ,V i , i∈(1,2,3……k), we get a total of 3 tensors Q,K,V, where the shapes of Q,K,V are all (k,256); Transform Q, K, V into tensors of shape (k, 8, 32), and further permute them into tensors PQ, PK, PV of shape (8, k, 32), where 8 is the number of heads in the multi-head attention layer and 32 is the dimension of the feature vector of each head; Decompose PQ into 8 matrices PQ of shape (k,32) i , i∈(1,2,3…8), decompose PK into 8 matrices PK of shape (k,32) i , i∈(1,2,3…8), 8 matrices PQ of shape (k,32) i Respectively with 8 matrices PK of shape (k,32) i Transpose and multiply to obtain 8 matrices of shape (k, k), which are divided by √32 to obtain 8 attention coefficient matrices attn_1 of shape (k, k); The relative distance between two human joints is defined as the number of bones connecting the two joints on the skeleton diagram, and this number of bones constitutes the shortest distance between the two joints. The relative distance matrix of a human skeleton diagram consisting of k joints is denoted by R, with a shape of (k, k), and its (i, j)th element represents the relative distance from the i-th joint to the j-th joint. Each relative distance value corresponds to an 8-dimensional weight vector, where 8 is the number of heads. Different relative distances correspond to different weight vectors. Each relative distance in R is used to query the corresponding weight vector to obtain a tensor of shape (k, k, 8), which is further permuted into a tensor of shape (8, k, k), denoted as RW. Decompose RW into 8 matrices of shape (k, k), which are divided by √32 to obtain the attention coefficient matrix attn_2; The obtained 8 attention coefficient matrices attn_1 and attn_2 are added to obtain 8 matrices of shape (k, k). The columns of the matrix are further softmaxed to obtain the final 8 attention coefficient matrices ATTN i , i∈(1,2,3…8); Decompose the obtained PV into 8 matrices PVi (i from 1 to 8) of shape (k, 32), and decompose 8 matrices ATTN of shape (k, k) i Multiply them with the transpose of 8 matrices of shape (k, 32) respectively to obtain 8 matrices of shape (k, 32). These 8 matrices of shape (k, 32) are transformed into a tensor of shape (8, k, 32), which is further permuted into a tensor of shape (k, 8, 32), and further transformed into a tensor of shape (k, 256). The obtained tensor is decomposed into k 256-dimensional feature vectors, and a transformation matrix is ​​input to obtain k 256-dimensional feature vectors.

3. A method for estimating 3D posture of a human body according to claim 1, characterized in that: The k 2D key point coordinates representing the human body posture are detected for the image to be recognized, where k is the predefined number of human body key points, including: By inputting the image to be identified into the MaskRCNN network, the coordinates of k 2D key points representing the human body posture are obtained.

4. A method for estimating 3D posture of a human body according to claim 1, characterized in that: The method further comprises: If there are multiple people in the image to be identified, the image will be cropped into a single person image before processing.

5. A human body 3D posture estimation device, characterized in that: The device comprises: 2D human posture detection module, which is used to input k 2D key point coordinates into a two-layer fully connected neural network, extract the 256-dimensional embedding feature vector corresponding to each key point coordinate, and output a total of k 256-dimensional embedding feature vectors; The 3D human posture estimation module is used to input the k 256-dimensional embedding feature vectors into the transformer encoder. The transformer encoder has a total of 4 encoder_layer layers. Each encoder_layer layer outputs k 256-dimensional vectors, and the output k 256-dimensional feature vectors are sequentially input into the next encoder_layer layer. The k 256-dimensional vectors output by the last encoder_layer layer are respectively input into two layers of fully connected neural networks. The output of the network is the 3D coordinates of each key point. The calculation process of each encoder_layer layer includes the following steps: a. Input k 256-dimensional feature vectors into the multi-head attention layer to obtain k 256-dimensional feature vectors; b. Then add the k 256-dimensional vectors obtained in step a to the k 256-dimensional feature vectors input by the encoder_layer layer to obtain k 256-dimensional feature vectors; c. Then perform LayerNorm operation on the k 256-dimensional feature vectors obtained in step b to obtain normalized k 256-dimensional vectors; d. Then, the k 256-dimensional vectors obtained in step c are input into a two-layer fully connected neural network to obtain k 256-dimensional vectors, where the k vectors share a fully connected feedforward network; e. Finally, add the k vectors obtained in step c and step d respectively, and perform LayerNorm operation on the added k vectors to obtain k normalized 256-dimensional vectors, and input these k 256-dimensional feature vectors into the next encoder_layer layer.

6. A human body 3D posture estimation system, characterized in that: The system includes: a processor and a memory; The memory is used to store one or more program instructions; The processor is used to run one or more program instructions to execute the method according to any one of claims 1-3.

7. A computer storage medium, characterized in that: The computer storage medium contains one or more program instructions, and the one or more program instructions are used by a human 3D posture estimation system to execute the method as described in any one of claims 1-3.

Images