3D Human Pose Estimation Method Based on Global and Local Spatiotemporal Encoders

Abstract

The present invention discloses a 3D human body posture estimation method based on global and local spatiotemporal encoders, and relates to the technical field of 3D human body posture estimation. The present invention adopts the coordinated design of global and local spatiotemporal encoders, which is used to extract 3D human body posture from monocular video, and the local spatiotemporal encoder includes four modules: a global spatial encoder, a global temporal encoder, a local spatiotemporal encoder, and a temporal aggregator, which can extract global and local spatiotemporal features, realize modeling and aggregation of the time series of human body posture, and add a spatiotemporal attention module to the local spatiotemporal encoder, which connects space and time attention in parallel and shares weights, and is used to model local spatiotemporal correlation. In this way, the local spatiotemporal information of joints can be better captured, the accuracy of posture estimation can be improved, and the spatiotemporal attention fusion module is used to fuse space and time attention to obtain more comprehensive and accurate spatiotemporal information.

Description

Technical Field

[0001] The present invention relates to the technical field of 3D human body posture estimation, and in particular to a 3D human body posture estimation method based on global and local spatiotemporal encoders. Background Art

[0002] With the development of computer technology, human-computer interaction technology is constantly being updated. Human-computer interaction technology based on vision provides people with a more flexible and convenient way of human-computer interaction by capturing and identifying body language, and has become a research hotspot in the field of artificial intelligence and natural human-computer interaction. The goal of 3D human posture estimation technology is to obtain the shape and movement of the human body in 3D space with the data collected by the sensor as input. Using the camera as a sensor and combining it with 3D human posture estimation technology based on deep learning can efficiently and accurately capture and identify human movements and body language, and this technology has been widely used in many fields, such as virtual reality, medical rehabilitation, and autonomous driving. But at the same time, 3DHPE faces more challenges. On the one hand, due to the difficulty of data collection, most of the current methods are based on monocular images or videos, and the mapping from 2D images to 3D postures is a multi-solution problem. On the other hand, deep learning algorithms rely on a large amount of training data, but due to the high difficulty and cost of 3D posture annotation, the current mainstream data sets are basically collected in laboratory environments, which is bound to affect the generalization performance of the algorithm on outdoor data.

[0003] The prior art has the following deficiencies:

[0004] (1) PoseFormer uses the self-attention mechanism of Transformer to describe the spatial dependencies between joints in each frame and the temporal dependencies between frames in the sequence. However, due to the similarity of human poses in adjacent frames in the video, there is a lot of redundant information in the entire time series, which requires a lot of computing resources. In addition, PoseFormer ignores the local context, but using local information is crucial in computer vision tasks.

[0005] (2) Strided Transformer proposed to model the long-term dependencies (global temporal correlations) of all joints of the human body through a common Transformer encoder architecture, and then model the temporal context information through the Strided Transformer encoder, gradually shortening the sequence length. However, they only obtain spatial information through a single linear layer, and the representation capability of this layer is insufficient.

[0006] (3) P-STMO captures the global spatial information of all joints in a single frame through the MLP block. At the same time, the temporal downsampling strategy and pre-training technology are adopted to reduce the difficulty of model training. This method models the temporal correlation of all joints of the human body, but does not consider the motion differences of each joint of the human body, resulting in insufficient learning of local spatiotemporal correlation.

[0007] Therefore, a new solution is needed to the above problem, and the applicant proposes a global and local spatiotemporal encoder to model spatiotemporal correlation. Specifically, a global spatial encoder and a global temporal encoder are constructed to capture the global spatial information of all joints in a single frame and the global temporal information of all frames, respectively, and a local spatiotemporal encoder is constructed to capture the spatiotemporal information of each joint in local N frames. In addition, the present invention proposes a parallel attention module with weight sharing to better merge spatial and temporal information into each node at the same time. The applicant believes that the model that simulates global and local spatiotemporal correlation is more suitable for 2D to 3D lifting tasks in videos. Summary of the invention

[0008] The purpose of the present invention is to provide a 3D human posture estimation method based on global and local spatiotemporal encoders to solve the technical problems raised in the background technology.

[0009] To achieve the above object, the present invention provides the following technical solutions: a 3D human body posture estimation method based on global and local spatiotemporal encoders, comprising a global spatial encoder, the global spatial encoder is used to fuse the spatial information of all joints of the human body in each frame, model the spatial information of a single posture through a fully connected residual network, learn sufficient spatial information from a 2D posture, and map it to a higher dimension;

[0010] A global temporal encoder, the global temporal encoder is used to compensate for the missing temporal information of the entire sequence of the global spatial encoder;

[0011] A local spatiotemporal encoder, the local spatiotemporal encoder is used to model the spatiotemporal correlation of each joint in a local frame;

[0012] The local spatiotemporal encoder includes a multi-head spatiotemporal self-attention mechanism module for simulating local spatiotemporal correlation;

[0013] The multi-head spatiotemporal self-attention mechanism module includes spatial attention Attn s and time attention Attn t ;

[0014] The local spatiotemporal encoder also includes a spatiotemporal attention fusion module, which is used to better integrate the spatial attention Attn s and time attention Attn tThe two branches of information complete the spatial attention Attn s and time attention Attn t The fusion of the two branches of information

[0015] The local spatiotemporal encoder also includes a network structure of LSTE;

[0016] A temporal aggregator, which is used to regress a single frame from the entire sequence to better regress the 3D pose of the intermediate frames and aggregate local information;

[0017] A global to local block is used to directly apply the global spatial encoder and the global temporal encoder to the local spatiotemporal encoder and the temporal aggregator.

[0018] Preferably, the global spatial encoder converts the 2D human posture sequence P∈R T×N×2 The 2D coordinates of all joints in each frame are combined as input The spatial information of each frame is embedded into high-dimensional features through the linear layer of the global spatial encoder where d g is the intermediate dimension of the global module to obtain a set of latent features as the input of the global temporal encoder, and the features of each frame are regarded as a separate token.

[0019] Preferably, the global temporal encoder is a Transformer encoder, which is composed of Layer composition;

[0020] Learnable temporal position embeddings Used before the first layer of the encoder, the formula is as follows:

[0021]

[0022] The global time encoder is applied to the input To get the output as follows:

[0023]

[0024]

[0025] in LN(·) is layer normalization;

[0026] The above formula can be expressed by the function GTE(·) as follows:

[0027]

[0028] Preferably, the multi-head spatiotemporal self-attention mechanism module includes three mapped learnable matrices Q, K and V;

[0029] Attention for space s and time attention Attn t matters in common;

[0030] Q s ,K s ,V s It is obtained by exchanging the dimensions of space and time;

[0031] Q t ,K t ,V t The complete sequence T is divided into two parts with local frame numbers The S sequences are obtained as follows:

[0032]

[0033]

[0034] Space Attention Attn s For modeling the spatial relationship between each joint in a single frame, temporal attention Attn t Used to simulate the temporal relationship between the same joints in a local T-frame;

[0035] Attn s and Attn t The statement is as follows:

[0036] Attn s (QK,V)=Reshape(Attention(Q s ,K s ,V s ))

[0037] Attn t (Q,K,V)=Concat(Attention(Q t ,K t ,V t ))

[0038] Among them, Attention(·) is the self-attention mechanism of Transformer, and spatial attention Attn s and time attention Attn t The shapes of the attention maps are T×N×N and

[0039] When MSA-ST has h attention heads, the MSA-ST operation is implemented by connecting all attention heads as follows:

[0040]

[0041] where i = 1, 2, ..., h, is a learnable matrix, is the spatiotemporal attention fusion module (STAF), Attn s (·) and Attn t The i-th head of (·).

[0042] Preferably, in the spatiotemporal attention fusion module, in the feature channel splicing space attention Attn s and time attention Attn t , the resulting dimension is

[0043] Compress the temporal dimension of input features through average pooling and maximum pooling operations;

[0044] This process generates two dimensions The features of the , and add the average feature and the maximum feature;

[0045] Finally, the features are passed through a fully connected network and a sigmoid operation to generate weights α s ,α t ∈

[0046] For space attention Attn s and time attention Attn t The fusion definition is as follows:

[0047] f cat =Concat(Attn s ,Attn t )

[0048] f avg+max =FC(AvgPool(f cat )+MaxPool(f cat ))

[0049]

[0050] α=Sigmoid(f′ avg+max )

[0051] α s =α[…:0],α t =α[…:1]

[0052]

[0053] Preferably, the input Z1 of the LSTE network structure is obtained through a global to local module, and the learnable spatial position embedding matrix It is used before the first layer of the local spatiotemporal encoder and its formula is as follows:

[0054]

[0055] The local features are modeled by combining the local spatiotemporal encoder and the temporal aggregator, and Layer composition;

[0056] The time dimension of each layer changes, so the model needs to embed the time position before each layer. The first layer of time position embedding is This process starts from the input To get the output The representation is as follows:

[0057]

[0058]

[0059]

[0060]

[0061] in TA(·) is the time aggregator, is the number of local frames in layer n, represents the time position embedded before the nth layer. The above formula can be expressed as follows:

[0062]

[0063] Preferably, the global to local block processes global features The global-to-local block implements global-to-local operations through projection and splitting, and is defined as follows:

[0064]

[0065] By exchanging the spatiotemporal channels of the 2D pose sequence P, we obtain the input X∈R of the local model N×T×2 Embedding;

[0066] The local embedding module maps the two-dimensional features of each node to a higher dimension d through linear projection. l ;

[0067] By combining To guide the global-to-local block projection from global to local, it is defined as follows:

[0068]

[0069] Among them, x i,j represents the i-th node of the j-th frame of X is the parameter matrix of the local embedding.

[0070] Preferably, the time aggregator aggregates the time information of each node through 1D convolution, and the input time series changes from T to S, where T and S represent the number of complete sequences of the current layer and the number of sequences after the current layer is segmented, respectively, expressed as a function It is the number of local frames of the segmentation sequence in the current layer, used in conjunction with the network structure of LSTE;

[0071] An additional temporal smoothness constraint is imposed by adding a regression head from the output of the global temporal encoder during training and obtaining the full sequence prediction output Y g ∈R T×(N·3) , add the regression head to the output of the LSTE network structure to obtain the predicted output Y l ∈R N×3 ;

[0072] Sequence loss is used to minimize the error between the predicted results and the true results:

[0073]

[0074] Where P i,j and denote the true and estimated 3D position sequences of joint i in the jth frame respectively;

[0075] Assuming that the network structure of LSTE models the local frames, the temporal aggregator aggregates the temporal information to finally obtain the features of the intermediate frames, and adopts the standard per-joint position error average loss to minimize the error between the predicted pose and the true pose, as shown below:

[0076]

[0077] Where P i and Y i are the true and estimated 3D positions of the i-th joint in the current pose;

[0078] The network is trained in an end-to-end manner, and the final loss function Defined as:

[0079]

[0080] where λg and λ l is the weighting factor.

[0081] In the pre-training stage, the global spatial encoder and the global temporal encoder are optimized so that the global spatial encoder and the global temporal encoder can learn the spatiotemporal correlation and inherent global features of the 2D joints by reconstructing the damaged 2D poses of randomly masked partial frames and partial 2D joints, thus providing a good initialization for the model.

[0082] In the fine-tuning phase, the weights trained in the pre-training phase are loaded into the global spatial encoder and the global temporal encoder, and then the entire model is trained to output the 3D joint positions at the full sequence scale and the single target frame scale;

[0083] The global space-time encoder and the local space-time encoder are used together. The input of the global space-time encoder and the local space-time encoder are 2D posture sequences after the temporal downsampling strategy, and the output is 3D joint positions at the full sequence and single target frame scale.

[0084] The global features are obtained by using a global spatial encoder to obtain the spatial information of all joints and a global temporal encoder to obtain the temporal information of all frames;

[0085] The spatiotemporal information of each joint in the local frame is obtained through the local spatiotemporal encoder, and new local features are obtained by aggregation through the temporal aggregator.

[0086] The 2D posture input in the pre-training stage and the fine-tuning stage adopts a temporal downsampling strategy TDS to alleviate data redundancy and expand the temporal receptive field.

[0087] Compared with the prior art, the present invention has the following beneficial effects:

[0088] 1. The present invention adopts the coordinated design of global and local spatiotemporal encoders to extract 3D human body posture from monocular video, and the local spatiotemporal encoder includes four modules: global spatial encoder, global temporal encoder, local spatiotemporal encoder and temporal aggregator, which can extract global and local spatiotemporal features and realize modeling and aggregation of time series of human body posture;

[0089] 2. The present invention adds a spatiotemporal attention module to the local spatiotemporal encoder. The module connects the spatial and temporal attention in parallel and shares the weights to model the local spatiotemporal correlation. In this way, the local spatiotemporal information of the joints can be better captured, the accuracy of posture estimation can be improved, and the spatiotemporal attention fusion module can be used to fuse the spatial and temporal attention to obtain more comprehensive and accurate spatiotemporal information.

[0090] 3. The present invention conducts two-stage training. In the pre-training stage, the GSE and GTE in the global module are mainly optimized. The spatiotemporal correlation and inherent global features of 2D joints are learned by reconstructing the damaged 2D postures of randomly masked partial frames and partial 2D joints, which provides a good initialization for the model. In the fine-tuning stage, the pre-trained weights are loaded into GSE and GTE, and then the entire model is trained, and the 3D joint positions at the full sequence scale and the single target frame scale are output. The 2D posture inputs in both stages adopt the time downsampling strategy. BRIEF DESCRIPTION OF THE DRAWINGS

[0091] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings required for describing the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying creative work.

[0092] Figure 1 It is the overall flow chart of the present invention;

[0093] Figure 2 This is a GLSTE example diagram for 3D pose estimation in the present invention;

[0094] Figure 3 A schematic diagram of the present invention consisting of multi-head self-attention in a global temporal encoder and multi-head spatiotemporal self-attention in a local spatiotemporal encoder;

[0095] Figure 4 Schematic diagram of the spatiotemporal attention fusion module of the present invention;

[0096] Figure 5 Schematic diagram of MPJPE metric comparison under different input frames in the Human3.6M test of the present invention;

[0097] Figure 6 Schematic diagram of qualitative comparison between the GLSTE of the present invention and humans in taking pictures, sitting and sitting down actions at 3.6M. DETAILED DESCRIPTION

[0098] The technical solutions in the embodiments of the present invention will be described clearly and completely below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than all the embodiments.

[0099] Embodiment 1:

[0100] See also Figure 1-4, a 3D human pose estimation method based on global and local spatiotemporal encoders, including a global spatial encoder, which is used to fuse the spatial information of all joints of the human body in each frame, model the spatial information of a single pose through a fully connected residual network, learn enough spatial information from 2D poses, and map them to higher dimensions, see Figure 3 (a);

[0101] Global time encoder,The global time encoder is used to make up for the missing time information of the entire sequence of the global spatial encoder, such as Figure 3 As shown in (b);

[0102] The movements of different joints of the human body are different, and the movement of the same joint in a few local frames is more obvious. Therefore, a local spatiotemporal encoder is used to simulate the spatiotemporal correlation of each joint in a local frame.

[0103] The local spatiotemporal encoder includes a multi-head spatiotemporal self-attention mechanism module for simulating local spatiotemporal correlations;

[0104] The multi-head spatiotemporal self-attention mechanism module includes spatial attention Attn s and time attention Attn t ;

[0105] The local spatiotemporal encoder also includes a spatiotemporal attention fusion module, which is used to better integrate the spatial attention Attn s and time attention Attn t The two branches of information complete the spatial attention Attn s and time attention Attn t The fusion of the two branches of information

[0106] The local spatiotemporal encoder also includes the network structure of LSTE;

[0107] Temporal Aggregator: The temporal aggregator is used to regress a single frame from the entire sequence to better regress the 3D pose of the intermediate frames and aggregate local information;

[0108] Global to Local Block,The global to local block is used to apply the global spatial encoder and the global temporal encoder directly to the local spatiotemporal encoder and temporal aggregator.

[0109] The global spatial encoder transforms the 2D human pose sequence P∈R T×N×2 The 2D coordinates of all joints in each frame are combined as input The spatial information of each frame is embedded into high-dimensional features through the linear layer of the global spatial encoder where d gis the intermediate dimension of the global module to obtain a set of latent features as the input of the global temporal encoder, and the features of each frame are regarded as a separate token.

[0110] The global temporal encoder is a Transformer encoder composed of Layer composition;

[0111] Learnable temporal position embeddings Used before the first layer of the encoder, the formula is as follows:

[0112]

[0113] The global time encoder is applied to the input To get the output as follows:

[0114]

[0115]

[0116] in LN(·) is the layer normalization, and the MSA in the global temporal encoder is as follows: Figure 3 As shown in (a), its attention matrix is ​​T×T;

[0117] The above formula can be expressed by the function GTE(·) as follows:

[0118]

[0119] The multi-head spatiotemporal self-attention mechanism module includes three mapped learnable matrices Q, K and V;

[0120] Attention for space s and time attention Attn t matters in common;

[0121] Q s ,K s ,V s It is obtained by exchanging the dimensions of space and time;

[0122] Q t ,K t ,V t The complete sequence T is divided into two parts with local frame numbers The S sequences are obtained as follows:

[0123]

[0124]

[0125] Space Attention Attn s For modeling the spatial relationship between each joint in a single frame, temporal attention Attn t Used to simulate the temporal relationship between the same joints in a local T-frame;

[0126] Attn s and Attn t The statement is as follows:

[0127] Attn s (QK,V)=Reshape(Attention(Q s ,K s ,V s ))

[0128] Attn t (Q,K,V)=Concat(Attention(Q t ,K t ,V t ))

[0129] Among them, Attention(·) is the self-attention mechanism of Transformer, and spatial attention Attn s and time attention Attn t The shapes of the attention maps are T×N×N and

[0130] When MSA-ST has h attention heads, the MSA-ST operation is implemented by connecting all attention heads as follows:

[0131]

[0132] where i = 1, 2, ..., h, is a learnable matrix, is the spatiotemporal attention fusion module (STAF), Attn s (·) and Attn t The i-th head of (·).

[0133] In the spatiotemporal attention fusion module, Figure 4 As shown. In the feature channel splicing space, attention is paid to Attn s and time attention Attn t , the resulting dimension is

[0134] Compress the temporal dimension of input features through average pooling and maximum pooling operations;

[0135] This process generates two dimensions The features of the , and add the average feature and the maximum feature;

[0136] Finally, the features are passed through a fully connected network and a sigmoid operation to generate weights α s ,α t ∈

[0137] For space attention Attn s and time attention Attn t The fusion definition is as follows:

[0138] f cat =Concat(Attn s ,Attn t )

[0139] f avg+max =FC(avgPool(f cat )+MaxPool(f cat ))

[0140]

[0141] α=Sigmoid(f′ avg+max )

[0142] α s =α[…:0],α t =α[…:1]

[0143]

[0144] The input Z1 of the LSTE network structure is obtained through the global to local module, and the learnable spatial position embedding matrix It is used before the first layer of the local spatiotemporal encoder and its formula is as follows:

[0145]

[0146] The network structure of LSTE models local features by combining a local spatiotemporal encoder and a temporal aggregator. Layer composition;

[0147] The time dimension of each layer changes, so the model needs to embed the time position before each layer. The first layer of time position embedding is This process starts from the input To get the output The representation is as follows:

[0148]

[0149]

[0150]

[0151]

[0152] in TA(·) is the time aggregator, is the number of local frames in layer n, represents the time position embedded before the nth layer. The above formula can be expressed as follows:

[0153]

[0154] Global to local blocks handle global features Given that the global feature is the feature of all nodes merged, and the local feature is the feature of a single node, the global-to-local block implements the global-to-local operation through projection and splitting, and its definition is as follows:

[0155]

[0156] By exchanging the spatiotemporal channels of the 2D pose sequence P, we obtain the input X∈R of the local model N×T×2 Embedding;

[0157] The local embedding module maps the two-dimensional features of each node to a higher dimension d through linear projection. l ;

[0158] By combining To guide the global-to-local block projection from global to local, it is defined as follows:

[0159]

[0160] Among them, x i,j represents the i-th node of the j-th frame of X is the parameter matrix of the local embedding.

[0161] Temporal Aggregator This task is to regress a single frame (intermediate frame) from the entire sequence. If only learning the temporal sequence without aggregation, the model will obtain a lot of redundant information and increase the computational overhead. Therefore, a temporal aggregator is proposed to better regress the 3D pose of the intermediate frame and aggregate local information. Specifically, we aggregate the temporal information of each node separately through 1D convolution, and the input temporal sequence changes from T to S, where T and S represent the number of complete sequences of the current layer and the number of sequences after the current layer is split, respectively, expressed as the function It is the number of local frames of the segmentation sequence in the current layer, which is used in conjunction with the network structure of LSTE;

[0162] An additional temporal smoothness constraint is imposed by adding a regression head from the output of the global temporal encoder during training and obtaining the full sequence prediction output Y g ∈R T×(N·3) , add the regression head to the output of the LSTE network structure to obtain the predicted output Y l ∈R N×3 ;

[0163] Since the output of the global temporal encoder is the feature information of all frames, the entire sequence scale needs to be monitored to achieve temporal consistency of the sequence. Therefore, the sequence loss is used to minimize the error between the predicted results and the true results:

[0164]

[0165] Where P i,j and denote the true and estimated 3D position sequences of joint i in the jth frame respectively;

[0166] Assuming that the network structure of LSTE models the local frames, the temporal aggregator aggregates the temporal information to finally obtain the features of the intermediate frames, and adopts the standard per-joint position error average loss to minimize the error between the predicted pose and the true pose, as shown below:

[0167]

[0168] Where P i and Y i are the true and estimated 3D positions of the i-th joint in the current pose;

[0169] The network is trained in an end-to-end manner, and the final loss function Defined as:

[0170]

[0171] where λ g and λ l is the weighting factor.

[0172] In the pre-training stage, the global spatial encoder and the global temporal encoder are optimized so that the global spatial encoder and the global temporal encoder can learn the spatiotemporal correlation and inherent global features of the 2D joints by reconstructing the damaged 2D poses of randomly masked partial frames and partial 2D joints, thus providing a good initialization for the model.

[0173] In the fine-tuning phase, the weights trained in the pre-training phase are loaded into the global spatial encoder and the global temporal encoder, and then the entire model is trained to output the 3D joint positions at the full sequence scale and the single target frame scale;

[0174] The global space-time encoder and the local space-time encoder are used together. The input of the global space-time encoder and the local space-time encoder are 2D posture sequences after the temporal downsampling strategy, and the output is 3D joint positions at the full sequence and single target frame scale.

[0175] The global features are obtained by using a global spatial encoder to obtain the spatial information of all joints and a global temporal encoder to obtain the temporal information of all frames;

[0176] The spatiotemporal information of each joint in the local frame is obtained through the local spatiotemporal encoder, and new local features are obtained by aggregation through the temporal aggregator.

[0177] The temporal downsampling strategy TDS is used for the 2D pose input in both the pre-training and fine-tuning stages to alleviate data redundancy and expand the temporal receptive field.

[0178] Embodiment 2:

[0179] This embodiment is used to further disclose a simulation application based on the above embodiment;

[0180] The model is evaluated on two 3D human pose estimation datasets: Human3.6M and MPIINF-3DHP.

[0181] Human3.6M is a widely used 3D human pose estimation dataset, which contains 3.6 million video frames and corresponding 3D human pose annotations. The dataset consists of 15 different activities of 11 actors, which are captured by four cameras with different viewpoints. The model is trained on five subjects S1, S5, S6, S7 and S8, and evaluated on two subjects S9 and S11 on 17 joint skeletons.

[0182] The most commonly used evaluation metrics MPJPE and PMPJPE are used.

[0183] MPJPE, i.e., protocol 1, is used to measure the average Euclidean distance between the joint positions of the predictions and the ground truth without any transformation.

[0184] P-MPJPE, i.e., Protocol 2, is MPJPE after aligning the predicted 3D pose with the ground truth using a rigid transformation.

[0185] Results on Human3.6M, comparing GLSTE with previous state-of-the-art methods on the Human360m dataset.

[0186] We use CPN as a 2D keypoint detector and then train the network on a sequence of detected 2D poses.

[0187] As shown in Table I, the performance of the models at 243 frames is shown below. Our method performs well with 42.2mm in Protocol 1 and 33.8mm in Protocol 2. We also use the same refinement module as UGCN to train our model, and it achieves 41.9mm under Protocol 1, outperforming all other methods.

[0188] As shown in Table II, when using the true 2D pose as input, our method achieves the best result 27.9mm,MPJPE, which outperforms all other methods.

[0189]

[0190] Table I uses the 2D pose detected by the Cascaded Pyramid Network (CPN) to quantitatively compare with the state-of-the-art methods on the Human3.6M dataset under Protocols 1 and 2. (*) The refinement module proposed in UGCN is used. The best results are shown in bold and the second best results are underlined.

[0191]

[0192] Table II quantitatively compares the MPJPE (mm) of the Human3.6M dataset under Scheme 1 with the ground truth 2D joint positions as input. The best results are shown in bold and the second best results are underlined.

[0193]

[0194] Table III quantitatively compares the state-of-the-art methods on different input frames on the Human 3.6M dataset. The MPJPE metric and number of parameters are reported, and the input 2D pose is detected by CPN.

[0195] In Table III, the comparison of the state-of-the-art methods in terms of total number of parameters, estimated floating point operations (FLOPs) per frame, and model performance for different input frames (T) is reported. It can be seen that the method in this case has the least parameters but has the best effect.

[0196] Compared to the methods VideoPose and Anatomy3D based on fully convolutional architectures, the Transformer architecture-based method P-STMO is more effective, and in this case the method outperforms the methods that also use pre-training and downsampling strategies.

[0197] It is worth noting that the results of the method with 27 frames as input can surpass the results of some recent SOTA methods with 81 frames and 243 frames as input, which shows the effectiveness and efficiency of the proposed method.

[0198] at the same time, Figure 5 The comparison of MPJPE metric under different input frames in the table is also shown.

[0199]

[0200]

[0201] Table IV Qualitative results of MPI-INF-3DHP

[0202] MPI-INF-3DHP is a challenging dataset for both indoor and outdoor scenes. It records 8 activities performed by 8 actors from 14 camera views, covering three scenes: green screen, non-green screen, and outdoor.

[0203] In the experiments, the metrics of MPJPE, percentage of correct keypoints within 150 mm (PCK), and area under the curve (AUC) within the PCK threshold range are reported as evaluation metrics.

[0204] Results on MPI-INF-3DHP. To evaluate the generalization ability of the proposed model, Table IV reports the quantitative results of GLSTE and other methods on MPI-INF-3DHP.

[0205] In this case, a 2D pose sequence of 81 frames is used as the input dataset to compare with Human3.6m. The table shows that the method in this case once again achieves state-of-the-art performance on all three evaluation metrics (PCK, AUC, and MPJPE). It emphasizes the effectiveness of GLSTE in improving the performance of outdoor scenes.

[0206] Qualitative results provide a visual comparison between the 3D estimated and true poses, and evaluate GLSTE on the Human3.6M test set S9 and S11 using three challenging actions (taking a photo, sitting, and sitting down). Compared with the state-of-the-art method P-STMO, the proposed method achieves more accurate predictions, such as Figure 6 shown.

[0207] It will be apparent to those skilled in the art that the invention is not limited to the details of the exemplary embodiments described above and that the invention can be implemented in other specific forms without departing from the spirit or essential features of the invention. Therefore, the embodiments should be considered exemplary and non-limiting in all respects, and the scope of the invention is defined by the appended claims rather than the foregoing description, and it is intended that all variations falling within the meaning and scope of the equivalent elements of the claims be included in the invention. Any reference numeral in a claim should not be considered as limiting the claim to which it relates.

Claims

1. A 3D human pose estimation method based on global and local spatiotemporal encoders, characterized by: It includes a global spatial encoder, which is used to fuse the spatial information of all joints of the human body in each frame, model the spatial information of a single posture through a fully connected residual network, learn sufficient spatial information from the 2D posture, and map it to a higher dimension; A global temporal encoder, the global temporal encoder is used to compensate for the missing temporal information of the entire sequence of the global spatial encoder; The global temporal encoder is a Transformer encoder; A local spatiotemporal encoder, the local spatiotemporal encoder is used to model the spatiotemporal correlation of each joint in a local frame; The local spatiotemporal encoder includes a multi-head spatiotemporal self-attention mechanism module for simulating local spatiotemporal correlation; The multi-head spatiotemporal self-attention mechanism module includes spatial attention Attn s and time attention Attn t ; The local spatiotemporal encoder also includes a spatiotemporal attention fusion module, which is used to better integrate the spatial attention Attn s and time attention Attn t The two branches of information complete the spatial attention Attn s and time attention Attn t The fusion of the two branches of information The local spatiotemporal encoder also includes a network structure of LSTE; The input Z1 of the LSTE network structure is obtained through the global to local module, and the learnable spatial position embedding matrix It is used before the first layer of the local spatiotemporal encoder and its formula is as follows: The local features are modeled by combining the local spatiotemporal encoder and the temporal aggregator, and Layer composition; The time dimension of each layer changes, so the model needs to embed the time position before each layer. The first layer of time position embedding is This process starts from the input To get the output The representation is as follows: in TA(·) is the time aggregator, is the number of local frames in layer n, represents the time position embedded before the nth layer, and the above formula is expressed as follows: A temporal aggregator, which is used to regress a single frame from the entire sequence to better regress the 3D pose of the intermediate frames and aggregate local information; A global to local block for applying a global spatial encoder and a global temporal encoder directly to a local spatiotemporal encoder and a temporal aggregator; MSA-ST has h attention heads. The MSA-ST operation is implemented by connecting all attention heads as follows: where i = 1, 2, ..., h, is a learnable matrix, is the spatiotemporal attention fusion module STAF, Attn s (·) and Attn t The i-th head of (·).

2. The 3D human body posture estimation method based on global and local spatiotemporal encoders according to claim 1, characterized in that: The global spatial encoder transforms the 2D human pose sequence P∈R T×N×2 The 2D coordinates of all joints in each frame are combined as input The spatial information of each frame is embedded into high-dimensional features through the linear layer of the global spatial encoder where d g is the intermediate dimension of the global module to obtain a set of latent features as the input of the global temporal encoder, and the features of each frame are regarded as a separate token.

3. The 3D human body posture estimation method based on global and local spatiotemporal encoders according to claim 2, characterized in that: The global temporal encoder is a Transformer encoder, which consists of Layer composition; Learnable temporal position embeddings Used before the first layer of the encoder, the formula is as follows: The global time encoder is applied to the input To get the output as follows: in LN(·) is layer normalization; The above formula is expressed by the function GTE(·) as follows:

4. The 3D human body posture estimation method based on global and local spatiotemporal encoders according to claim 3, characterized in that: The multi-head spatiotemporal self-attention mechanism module includes three mapped learnable matrices Q, K and V; Attention for space s and time attention Attn t matters in common; Q s ,K s ,V s It is obtained by exchanging the dimensions of space and time; Q t ,K t ,V t The complete sequence T is divided into two parts with local frame numbers The S sequences are obtained as follows: Space Attention Attn s For modeling the spatial relationship between each joint in a single frame, temporal attention Attn t Used to simulate the temporal relationship between the same joints in a local T-frame; Attn s and Attn t The statement is as follows: Attn s (QK,V)=Reshape(Attention(Q s ,K s ,V s )) Attn t (Q,K,V)=Concat(Attention(Q t ,K t ,V t )) Among them, Attention(·) is the self-attention mechanism of Transformer, and spatial attention Attn s and time attention Attn t The shapes of the attention maps are T×N×N and 5. The 3D human body posture estimation method based on global and local spatiotemporal encoders according to claim 4, characterized in that: In the spatiotemporal attention fusion module, the feature channel splicing space attention Attn s and time attention Attn t , the resulting dimension is Compress the temporal dimension of input features through average pooling and maximum pooling operations; This process generates two dimensions The features of the , and add the average feature and the maximum feature; Finally, the features are passed through a fully connected network and a sigmoid operation to generate weights For space attention Attn s and time attention Attn t The fusion definition is as follows: f cat =Concat(Attn s ,Attn t ) f avg+max =FC(AvgPool(f cat )+MaxPool(f cat )) α=Sigmoid(f′ avg+max ) α s ＝α[…:0],α t ＝α[…:1] 6. The 3D human body posture estimation method based on global and local spatiotemporal encoders according to claim 5, characterized in that: The global to local block processes global features The global-to-local block implements global-to-local operations through projection and splitting, and is defined as follows: By exchanging the spatiotemporal channels of the 2D pose sequence P, we obtain the input X∈R of the local model N×T×2 Embedding; The local embedding module maps the two-dimensional features of each node to a higher dimension d through linear projection. l ; By combining To guide the global-to-local block projection from global to local, it is defined as follows: Among them, x i,j represents the i-th node of the j-th frame of X is the parameter matrix of the local embedding.

7. The 3D human body posture estimation method based on global and local spatiotemporal encoders according to claim 6, characterized in that: The time aggregator aggregates the time information of each node through 1D convolution. The input time series changes from T to S, where T and S represent the number of complete sequences of the current layer and the number of sequences after the current layer is split, respectively, expressed as a function is the local frame number of the segmentation sequence in the current layer, which is used in conjunction with the network structure of LSTE.

8. The 3D human pose estimation method based on global and local spatiotemporal encoders according to claim 7, characterized in that: At least the following steps are included: In the pre-training stage, the global spatial encoder and the global temporal encoder are optimized so that the global spatial encoder and the global temporal encoder can learn the spatiotemporal correlation and inherent global features of the 2D joints by reconstructing the damaged 2D poses of randomly masked partial frames and partial 2D joints, thus providing a good initialization for the model. In the fine-tuning phase, the weights trained in the pre-training phase are loaded into the global spatial encoder and the global temporal encoder, and then the entire model is trained to output the 3D joint positions at the full sequence scale and the single target frame scale; The global space-time encoder and the local space-time encoder are used together. The input of the global space-time encoder and the local space-time encoder are 2D posture sequences after the temporal downsampling strategy, and the output is 3D joint positions at the full sequence and single target frame scale. The global features are obtained by using a global spatial encoder to obtain the spatial information of all joints and a global temporal encoder to obtain the temporal information of all frames; The spatiotemporal information of each joint in the local frame is obtained through the local spatiotemporal encoder, and new local features are obtained by aggregation through the temporal aggregator.

9. The 3D human body posture estimation method based on global and local spatiotemporal encoders according to claim 8, characterized in that: The 2D posture input in the pre-training stage and the fine-tuning stage adopts a temporal downsampling strategy TDS to alleviate data redundancy and expand the temporal receptive field.

Images