Multi-view 3D pose estimation system based on Transformer

Abstract

The present invention belongs to the field of computer vision, and specifically relates to a Transformer-based multi-view 3D posture estimation system. The system is divided into two parts according to the processing flow sequence: a single-view spatiotemporal Transformer feature extraction module and a multi-view Transformer feature exchange module. The single-view spatiotemporal Transformer feature extraction module extracts spatiotemporal feature information of different perspectives from the input multi-view posture sequence; the multi-view Transformer feature exchange module exchanges the spatiotemporal feature information of different perspectives in pairs, and then outputs the final 3D posture estimation result through a regression head. The present invention can make full use of the temporal context and spatial relationship between posture sequences, fully exchange and transmit features between different perspectives, and improve the accuracy of the 3D posture estimation result.

Description

Technical Field

[0001] The present invention belongs to the field of computer vision technology, and in particular relates to a Transformer-based multi-view 3D posture estimation system, which has good performance in 3D posture estimation tasks. Background Art

[0002] 2D-3D pose estimation is a popular research direction in the field of 3D pose estimation and is the basis for upper-level tasks (such as pedestrian re-identification, human-computer interaction, behavior recognition, etc.). 3D pose estimation predicts the 3D coordinates of each key point of the human body based on the 2D coordinates of the key points of the human body. The main problem in the field of 2D-3D pose estimation is depth ambiguity, which means that since the 2D skeleton has lost depth information, it is difficult for the pose estimation model to infer the true distance of each key point in the 2D skeleton from the lens. At present, some work uses video to solve the problem of depth ambiguity, using adjacent frames to predict the 3D coordinates of human key points from the 2D skeleton sequence, but this implicit method of extracting spatiotemporal relationships has low accuracy. Some work uses multi-view images to explicitly predict depth information, but this multi-view-based method usually requires a lot of computing resources to construct a three-dimensional space representation, which is difficult to apply in practice and further optimize. Summary of the invention

[0003] In order to solve the problems existing in the prior art, the present invention proposes a multi-view 3D pose estimation system based on the spatiotemporal Transformer module, which can improve the accuracy and speed of the 3D pose estimation task.

[0004] The Transformer-based multi-view 3D pose estimation system proposed in this invention is divided into two parts according to the processing sequence: a single-view spatiotemporal Transformer feature extraction module and a multi-view Transformer feature exchange module. The schematic diagram of the entire system is shown in the figure. Figure 1 shown.

[0005] The 2D skeleton sequences of different perspectives are respectively input into the single-view spatiotemporal Transformer feature extraction module to extract the spatiotemporal feature information of different perspectives; the multi-view Transformer feature exchange module exchanges the spatiotemporal feature information of different perspectives in pairs, and then outputs the final 3D posture estimation result through the regression head.

[0006] The following is a brief introduction to the system contents:

[0007] 1. Single-view spatiotemporal Transformer feature extraction module

[0008] Single-view Spatial and Temporal Transformer feature extraction module (Single-view Spatial and Temporal Transformer, S-ST 2 ) consists of two parts: posture and position embedding layer and nested Transformer block. Its network structure is as follows Figure 2 shown.

[0009] (1) Posture and position embedding layer

[0010] The pose and position embedding layers embed joints and poses into word tokens and sentence tokens respectively, which are the precursors of the nested Transformer blocks. Given an input pose sequence of T frames Each frame P i Contains the 2D coordinates J of N joints i,k Using R n To represent the feature space with dimension n, we have J i,k ∈R 2 , i∈1,2,...,T, k∈1,2,...,N; in order to construct a word token and sentence tokens Use linear layers separately and As pose embedding and position embedding, where d w and d s are the feature dimensions of words and sentences respectively. The construction process can be expressed by formulas (1) and (2):

[0011]

[0012]

[0013] In the formula and are the weights and biases of the input embeddings for words and sentences, respectively. and denote all words and all sentences in the i-th sentence respectively, ∥ denotes channel-by-channel concatenation, and the position embeddings of words and sentences explicitly encode indexes in spatial and temporal dimensions.

[0014] (2) Nested Transformer blocks

[0015] The nested Transformer block receives word tokens from the pose and position embedding layers. and sentence tokens This step uses a series of nested Transformer blocks to extract and transfer features from tokens. Linear transformation and element-by-element accumulation are applied between the outer and inner sub-blocks of each nested Transformer block to transfer the spatial relationship to the sentence token. The feature extraction and transfer process of the nested Transformer block can be expressed by formulas (3), (4) and (5);

[0016]

[0017]

[0018]

[0019] In the formula and They represent the weight and bias of the linear transformation respectively, l represents the nested layer number, and ∥ represents channel-by-channel cascading.

[0020] After a series of nested Transformer blocks, the monocular 3D pose sequence inferred by the regression head is used as intermediate supervision. The regression head consists of a global average pooling layer, a normalization layer, and a linear transformation layer.

[0021] 2. Multi-view Transformer feature exchange module

[0022] The multi-view Transformer feature exchange module consists of multiple cross-view Transformer feature transmission sub-modules (Cross-view Transformer-based Transmission modules, CT 2 ), and its network structure is as follows Figure 3 shown.

[0023] The encoder and decoder of the cross-view Transformer feature transfer submodule each accept a sequence as input and use the decoder output as the final output. The encoder and decoder are stacked by Transformer sub-blocks, and a guided attention mechanism (GA) is added between the sub-blocks to find highly relevant key point pairs between different views. The guided attention is at the encoder output Y∈R n×d and decoder input X∈R n×d Perform Multi-head Attention (MHA) between:

[0024] GA(X,Y)=MHA(FC Q (X),FC K (Y),FC V (Y)) (6)

[0025] SubBlock G (X,Y)=LN(X+GA(X,Y)) (7)

[0026] Therefore, the codec structure can be expressed by formula (8):

[0027] ED(X,Y)=Decoder(X+Encoder(Y)) (8)

[0028] CT 2 The main idea of ​​guiding the features between the two views in parallel can be expressed by formula (9) and formula (10):

[0029] CT 2 (X,Y) X =ED(X,Y) (9)

[0030] CT 2 (X,Y) X =ED(Y,X) (10)

[0031] The multi-view Transformer feature exchange module divides all views into groups of two. groups, where V represents the number of viewing angles, and each group uses CT 2 The module transfers feature information. This module performs V-1 groupings in total to ensure that any two views have exchanged features. After the exchange, the feature information of different views passes through a series of regression heads to obtain the final 3D pose estimation result.

[0032] Compared with the prior art, the present invention has the following beneficial effects:

[0033] The system of the present invention can make full use of the temporal context and spatial relationship between posture sequences, fully exchange and transmit features between different perspectives, and improve the accuracy of 3D posture estimation results. The system proposed by the present invention performs better than existing methods in 3D posture estimation tasks. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] Figure 1 : Schematic diagram of a multi-view 3D pose estimation system based on Transformer.

[0035] Figure 2 : Schematic diagram of the network structure of the single-view spatiotemporal Transformer feature extraction module.

[0036] Figure 3 : Schematic diagram of the network structure of the multi-view Transformer feature exchange module. DETAILED DESCRIPTION

[0037] In the 3D pose estimation task, this paper (a multi-view 3D pose estimation system based on Transformer, ST 2 PE) and baseline methods were compared using three pose estimation datasets: Human3.6M, HumanEva, and MPI-INF-3DHP. The Human3.6M dataset is the largest and most widely used 3D human pose dataset, with 3.6 million video frames from 11 professional actors, including 17 different actions from 4 synchronized cameras. The HumanEva dataset has a total of 70,000 video frames collected by a motion capture system, with 4 actors performing 6 common actions. The MPI-INF-3DHP dataset contains actions performed by 8 actors from 14 perspectives.

[0038] The experimental environment is as shown in Table 1.

[0039] Table 1 Experimental environment

[0040] name illustrate processor CPU+GPU Graphics NVIDIA RTX 3090 System Type Ubuntu 18.04 (64-bit) Installing Memory 32G Python 3.7 Pytorch 1.0 Cuda 10.0

[0041] The present invention and the baseline method are compared in posture estimation experiments on three datasets: Human3.6M, HumanEva, and MPI-INF-3DHP. The results of the comparative experiments are shown in Tables 2, 3, and 4.

[0042] Table 2 MPJPE and P-MPJPE of pose estimation methods on Human3.6M dataset

[0043] method MPJPE P-MPJPE Tom et al.[1] 52.8 44.6 Cai et al.[2] 48.8 39.0 Wang et al.[3] 45.6 35.6 Zheng et al.[4] 44.3 34.6 Li et al.[5] 43.6 34.9 Ours 32.0 23.9

[0044] Table 3 MPJPE and P-MPJPE of pose estimation methods on HumanEva dataset

[0045] method MPJPE P-MPJPE PoseFormer[4] 37.9 31.2 Ours 22.4 17.6

[0046] Table 4 MPJPE and P-MPJPE of pose estimation methods on the MPI-INF-3DHP dataset

[0047]

[0048]

[0049] According to the results in Tables 1, 2, 3 and 4, the proposed system performs better than existing methods in 3D pose estimation tasks.

[0050] [1]Tome,et al."Rethinking pose in 3d:Multi-stage refinement andrecovery for markerless motion capture".2018international conference on 3Dvision.2018.

[0051] [2]Cai,et al."Exploiting spatial-temporal relationships for 3d poseestimation via graph convolutional networks".Proceedings of the IEEE / CVFinternational conference on computer vision.2019.

[0052] [3]Wang,et al."Motion guided 3d pose estimation from videos".Europeanconference on computer vision.2020.

[0053] [4]Zheng,et al."3D human pose estimation with spatial and temporaltransformers".Proceedings of the IEEE international conference on computervision.2021.

[0054] [5]Li,et al."Lifting transformer for 3d human pose estimation invideo".arXiv preprint arXiv:2103.14304.2021。

Claims

1. A multi-view 3D pose estimation system based on Transformer module, characterized in that: It includes several It consists of two parts: single-view spatiotemporal Transformer feature extraction module and multi-view Transformer feature exchange module; The single-view spatiotemporal Transformer feature extraction module extracts spatiotemporal feature information from different viewpoints of the input multi-view 2D skeleton sequence; The multi-view Transformer feature exchange module exchanges the spatiotemporal feature information of different views in pairs, and then outputs the final multi-view 3D pose estimation result through the regression head; The single-view spatiotemporal Transformer feature extraction module consists of two parts: a posture and position embedding layer and a nested Transformer block. The posture and position embedding layer embeds joints and postures into word tokens and sentence tokens respectively, and the nested Transformer block receives word tokens from the posture and position embedding layers. and sentence tokens Extract and transfer features from word tokens and sentence tokens; (1) Posture and position embedding layer Given an input pose sequence of T frames Each frame P i Contains the 2D coordinates J of N joints i,k ; Using R n To represent the feature space with dimension n, we have J i,k ∈R 2 , i∈1,2,...,T, k∈1,2,...,N; in order to construct a word token and sentence tokens Use linear layers separately and As pose embedding and position embedding, where d w and d s are the feature dimensions of words and sentences respectively; the construction process of word tokens and sentence tokens is expressed by formulas (1) and (2): In the formula and are the weights and biases of the input embeddings for words and sentences, respectively; and Represent all words and all sentences in the i-th sentence respectively, and ∥ represents channel-by-channel concatenation; the position embeddings of words and sentences explicitly encode indexes in spatial and temporal dimensions; (2) Nested Transformer blocks Get word token and sentence tokens Finally, a series of nested Transformer blocks are applied to these tokens to extract and transfer features. Linear transformation and element-by-element accumulation are applied between the outer and inner sub-blocks of each nested Transformer block to transfer the spatial relationship to the sentence token. The feature extraction and transfer process of the nested Transformer block is expressed by formulas (3), (4) and (5): In the formula and Respectively represent the weight and bias of linear transformation, l represents the nesting layer number, and ∥ represents channel-by-channel cascading; The multi-view Transformer feature exchange module consists of multiple cross-view Transformer feature transfer sub-modules CT 2 composition; The encoder and decoder of the cross-view Transformer feature transfer submodule each accept a sequence as input and use the decoder output as the final output; the encoder and decoder are respectively stacked by Transformer sub-blocks, and an additional guided attention GA mechanism is added between the Transformer sub-blocks to find highly correlated key point pairs between different views; the guided attention is at the encoder output Y∈R n×d and decoder input X∈R n×d Perform multi-head attention MHA between: GA(X,Y)=MHA(FC Q (X),FC K (Y),FC V (Y)) (6) SubBlock G (X,Y)=LN(X+GA(X,Y)) (7) The structure of the codec is expressed by formula (8): ED(X,Y)=Decoder(X+Encoder(Y)) (8) Cross-view Transformer feature transfer submodule CT 2 The features between the two views are guided in parallel, expressed by formula (9) and formula (10): CT 2 (X,Y) X =ED(X,Y) (9) CT 2 (X,Y) X =ED(Y,X) (10) The multi-view Transformer feature exchange module divides all views into groups of two. group, V represents the number of viewing angles, and each group uses CT 2 The module is used to transmit feature information; this module performs V-1 groupings in total to ensure that features are exchanged between any two perspectives.

2. The multi-view 3D posture estimation system according to claim 1, characterized in that: In the single-view spatiotemporal Transformer feature extraction module, after a series of nested Transformer blocks, the single-view 3D pose sequence inferred by the regression head is used as intermediate supervision.

3. The multi-view 3D posture estimation system according to claim 1 or 2, characterized in that: The regression head consists of a global average pooling layer, a normalization layer, and a linear transformation layer.

Images