Three-dimensional human body posture estimation method and device, electronic equipment and medium

Abstract

The invention provides a three-dimensional human body posture estimation method, which comprises the following steps of: acquiring a human body posture estimation data set, and preprocessing the data set; and obtaining a continuous two-dimensional attitude sequence from the data set through a two-dimensional attitude detector, and inputting the continuous two-dimensional attitude sequence into the STGFormer to obtain a three-dimensional attitude of continuous frames. According to the embodiment of the invention, the STGFormer architecture is provided, and the architecture adopts a unique'first trimming and then recovery 'strategy, that is, only part of key attitude tokens are selected as the input of the STGFormer block in the processing process, and the full-length sequence is recovered before final output, so that the calculation efficiency is remarkably improved, and the unnecessary calculation overhead is reduced.

Description

Technical Field

[0001] The present invention relates to the field of deep learning and computer vision technology, and in particular to a three-dimensional human body posture estimation method, device, electronic equipment and medium. Background Art

[0002] 3D human pose estimation, as an important technology in the field of computer vision, is based on reconstructing the three-dimensional pose of the human body from two-dimensional images or video data, that is, determining the position coordinates of each joint of the human body in three-dimensional space. This technology has shown important value in many fields such as human-computer interaction, motion analysis, and behavior recognition due to its wide application prospects. However, 3D human pose estimation also faces many technical challenges.

[0003] The primary difficulty lies in the ambiguity in the mapping process from 2D images to 3D poses. Since 2D images themselves lack depth information, one 2D pose may correspond to multiple possible 3D poses, which increases the difficulty of 3D pose estimation. In addition, the occlusion problem of the human body in the image and the lack of large-scale, diverse outdoor datasets are also obstacles that need to be overcome in this field.

[0004] Related technologies use the temporal information contained in video sequences to enhance the accuracy and robustness of 3D human pose estimation. By analyzing the changes between consecutive frames, the 3D pose of the human body can be better inferred.

[0005] In recent years, the rise of Transformer models in the field of computer vision has brought new breakthroughs in 3D human pose estimation. Although Transformer is good at modeling the long-range dependencies of input tokens, it is less efficient in capturing local information, while Graph Convolutional Networks (GCNs) are good at processing local dependencies but lack the ability to capture long-range dependencies. Related technologies mainly rely on Transformer or GCN-based architectures, which are not the best choice for 3D human pose estimation.

[0006] Although Transformer-based methods have achieved significant progress in performance, their high computational cost cannot be ignored. Especially when processing long video sequences, the computational complexity of the self-attention mechanism in Transformer rises sharply with the number of tokens, which limits the application of this method on edge devices with limited computing resources. In addition, due to the high similarity between adjacent video frames, traditional Transformer models often contain a lot of redundant information when processing, further reducing computational efficiency. Summary of the invention

[0007] In view of this, the embodiments of the present disclosure provide a 3D human posture estimation method, device, electronic device and medium, which at least partially solve the problems existing in the prior art.

[0008] In a first aspect, an embodiment of the present disclosure provides a 3D human body posture estimation method, which includes:

[0009] Acquire a human body posture estimation data set, and preprocess the data set;

[0010] Obtaining a continuous two-dimensional pose sequence from the data set through a two-dimensional pose detector, and inputting the continuous two-dimensional pose sequence into STGFormer to obtain three-dimensional poses of continuous frames;

[0011] The three-dimensional poses of consecutive frames obtained by the STGFormer include:

[0012] Using the continuous 2D pose sequence with J joints and T frames As input, where the input channel size is 2;

[0013] By linear embedding P 2D Projection to high-dimensional features The feature dimension of each joint representation is d;

[0014] Embedding spatial locations Add to the token to get a 2D pose sequence containing joint position information, where

[0015] Using L STGFormer blocks to learn spatiotemporal context from the 2D pose sequence containing joint position information to obtain a 2D pose sequence containing a comprehensive motion representation of the human body;

[0016] Selecting the 2D pose sequence containing the overall motion representation of the human body using a token pruning module to obtain a 2D pose sequence containing high semantic diversity tokens;

[0017] The 2D pose sequence containing high semantic diversity tokens is restored to a full-length sequence through a recovery head, and a 3D pose sequence is estimated from the full-length sequence

[0018] According to a specific implementation of the embodiment of the present disclosure, the STGFormer block includes: a spatial Graph Transformer and a temporal Graph Transformer;

[0019] The spatial Graph Transformer processes each body joint as a different token to learn the body joint relationship within each frame;

[0020] The Temporal Graph Transformer treats each frame as a single token to learn the temporal correlations between frames.

[0021] According to a specific implementation of the embodiment of the present disclosure, the processing of the spatial Graph Transformer includes:

[0022] Using spatial multi-head self-attention and spatial graph convolutional networks to learn the spatial correlation of all joints in a single frame

[0023] The formula for modeling is as follows:

[0024] X′ s =X s +GCN s (LN(X s ))

[0025] X″ s =X′ s +MSA s (LN(X′ s ))

[0026] X″′ s =X″ s +MLP(LN(X″ s ))

[0027] in, represents the spatial features of the input, LN represents layer normalization, GCN s Represents the spatial GCN, the adjacency matrix is ​​represented by the topological matrix of the human skeleton, MSA s stands for Spatial MSA and MLP stands for Multilayer Perceptron.

[0028] According to a specific implementation of the embodiment of the present disclosure, the processing of the time Graph Transformer includes:

[0029] The spatial features output by the spatial Graph Transformer As input;

[0030] Will Transformed into From the time dimension, different motion trajectories of human joints are modeled through temporal multi-head self-attention and temporal graph convolutional network. The modeling formula is as follows:

[0031] X′ T =X T +GCN T (LN(X T ))

[0032] X″ T =X′ T +MSA T (LN(X′ T ))

[0033] X″′ T =X″ T +MLP(LN(X″ T ))

[0034] Among them, LN stands for layer normalization, MSA T Representing time MSA, GCN T stands for Temporal GCN and MLP stands for Multi-Layer Perceptron.

[0035] According to a specific implementation of an embodiment of the present disclosure, the token pruning module processing includes:

[0036] For input gesture tokens Spatial redundancy is eliminated by averaging spatial pooling to obtain spatial pooling tokens

[0037] Randomly select t tokens X j ∈X i As the initial cluster center,

[0038] The distances of other tokens from the cluster center are then calculated and assigned to the nearest cluster as follows:

[0039] d(X i , X j )=‖X i -X j || 2 ,

[0040] For each cluster center X j , calculate all tokens X in the cluster to which it belongs i Total distance to cluster center:

[0041]

[0042] Among them, C j Represents the cluster center X j The cluster to which it belongs;

[0043] Select new cluster centers The total distance s from all tokens in the cluster to the cluster center j Minimum:

[0044]

[0045] After the iteration is completed, the cluster center of each cluster is the final cluster center, and the cluster center is selected as the representative token And other tokens are discarded to reduce redundant information in the video sequence.

[0046] According to a specific implementation of the embodiment of the present disclosure, the recovery head is composed of an MCA and a linear regression head;

[0047] The processing steps of the MCA are:

[0048] The learnable tokens that will be initialized to zero As Q, the representative token output by the last STGFormer block As K and V, and then perform residual connection as follows:

[0049]

[0050] Among them, query key Sum d represents the dimension, {n q , n k , n v} are the number of tokens of {Q, K, V} respectively, is the recovered token, whose time dimension is the same as the learnable token same;

[0051] Finally, the 3D pose coordinates are estimated using the linear regression head

[0052] According to a specific implementation of the embodiment of the present disclosure, the processing of the multi-head attention mechanism includes: input feature Linear Mapping to Query key Sum Where n represents the sequence length and d represents the dimension of the input feature;

[0053] Compute the token-to-token association matrix and pass information between different tokens as follows:

[0054]

[0055] Split the query, key, and value into H times and perform attention in parallel, and concatenate the outputs of the H attention heads;

[0056] The processing of graph convolutional networks includes: extracting features, as follows:

[0057]

[0058] in, represents the adjacency matrix, express The sum of the elements along the diagonal, W1 and W2 represent the weight matrices, BN represents batch normalization, and σ represents the ReLU activation function.

[0059] In a second aspect, an embodiment of the present disclosure provides a 3D human body posture estimation device, comprising:

[0060] A linear embedding module is used to map the original 2D pose coordinates from the two-dimensional space to the high-dimensional feature space;

[0061] A spatial position embedding module is used to embed the spatial position into the token to obtain a 2D pose sequence containing joint position information;

[0062] The STGFormer module includes a spatial Graph Transformer and a temporal Graph Transformer, which is used to capture spatial and temporal features in human posture data;

[0063] A token pruning module to select representative tokens for subsequent processing to obtain a 2D pose sequence containing representative tokens with high semantic diversity;

[0064] The recovery header module is used to restore the pruned tokens to the original full-length sequence size;

[0065] The 3D pose estimation module is used to estimate the 3D pose sequence from the restored full-length sequence.

[0066] In a third aspect, an embodiment of the present disclosure provides an electronic device, the electronic device comprising:

[0067] at least one processor; and,

[0068] a memory communicatively connected to the at least one processor; wherein,

[0069] The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the above-mentioned three-dimensional human body posture estimation method.

[0070] In a fourth aspect, an embodiment of the present disclosure provides a non-transitory computer-readable storage medium, which stores computer instructions for causing the computer to execute the above-mentioned three-dimensional human body posture estimation method.

[0071] In summary, compared with the prior art, this embodiment has the following advantages:

[0072] The STGFormer architecture provided in this embodiment combines the global modeling capabilities of Transformer with the advantages of graph convolutional networks in capturing local spatiotemporal relationships. It not only enhances the model's ability to learn complex spatiotemporal correlations between human joints, but also improves the accuracy and robustness of 3D human posture estimation.

[0073] In order to ensure high performance of the model while effectively controlling computational complexity and resource consumption, pose sequence pruning was introduced. By identifying and removing highly similar redundant pose frames in the video sequence, the amount of data processing was effectively reduced, thereby accelerating the training and inference process and improving overall efficiency.

[0074] The integration of spatiotemporal self-attention mechanism and spatiotemporal GCN enables the model to fully and deeply understand the dynamic changes of human posture. In order to further optimize the performance and efficiency of the model, a token pruning module is designed. This module evaluates the importance of the posture of each frame in the video sequence. By eliminating those posture frames with low information contribution and high semantic repetitiveness, the module successfully constructs a streamlined and efficient posture token set, providing more efficient and high-quality input data for subsequent posture estimation tasks. BRIEF DESCRIPTION OF THE DRAWINGS

[0075] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with specific implementation methods and drawings. Here, the exemplary implementation methods of the present invention and their descriptions are used to explain the present invention, but are not intended to limit the present invention.

[0076] As used herein, the term "including" and its variations mean open inclusion, i.e., "including but not limited to". Unless otherwise stated, the term "or" means "and / or". The term "based on" means "based at least in part on". The terms "an example embodiment" and "an embodiment" mean "at least one example embodiment". The term "another embodiment" means "at least one additional embodiment". The terms "first", "second", etc. may refer to different or the same objects. Other explicit and implicit definitions may also be included below.

[0077] Figure 1 A schematic diagram of a flow chart of a 3D human body posture estimation method provided by the first embodiment of the present invention;

[0078] Figure 2 A schematic diagram of the structure of STGFormer;

[0079] Figure 3 A schematic diagram of the structure of the token pruning module;

[0080] Figure 4 A schematic diagram of the structure of a three-dimensional human posture estimation device provided in the second embodiment of the present invention.

[0081] Figure 5 An exemplary structural diagram of a device capable of implementing the method according to an embodiment of the present invention is shown. DETAILED DESCRIPTION

[0082] The embodiments of the present disclosure are described in detail below with reference to the accompanying drawings.

[0083] The following describes the embodiments of the present disclosure through specific examples, and those skilled in the art can easily understand other advantages and effects of the present disclosure from the contents disclosed in this specification. Obviously, the described embodiments are only a part of the embodiments of the present disclosure, rather than all of the embodiments. The present disclosure can also be implemented or applied through other different specific embodiments, and the details in this specification can also be modified or changed in various ways based on different viewpoints and applications without departing from the spirit of the present disclosure. It should be noted that the following embodiments and features in the embodiments can be combined with each other without conflict. Based on the embodiments in the present disclosure, all other embodiments obtained by ordinary technicians in the field without making creative work are within the scope of protection of the present disclosure.

[0084] It should be noted that various aspects of the embodiments within the scope of the appended claims are described below. It should be apparent that the aspects described herein may be embodied in a wide variety of forms, and any specific structure and / or function described herein is merely illustrative. Based on the present disclosure, it should be understood by those skilled in the art that an aspect described herein may be implemented independently of any other aspect, and two or more of these aspects may be combined in various ways. For example, any number of aspects described herein may be used to implement the device and / or practice the method. In addition, other structures and / or functionalities other than one or more of the aspects described herein may be used to implement this device and / or practice this method.

[0085] It should also be noted that the illustrations provided in the following embodiments are only schematic illustrations of the basic concept of the present disclosure. The drawings only show components related to the present disclosure rather than being drawn according to the number, shape and size of components in actual implementation. In actual implementation, the type, quantity and proportion of each component may be changed arbitrarily, and the component layout may also be more complicated.

[0086] Additionally, in the following description, specific details are provided to facilitate a thorough understanding of the examples. However, it will be understood by those skilled in the art that the aspects described may be practiced without these specific details.

[0087] See also Figure 1The first embodiment of the present invention provides a 3D human body posture estimation method, which can be executed by an electronic device, and in particular, by one or more processors in the electronic device, to implement the following steps:

[0088] S101, obtaining a human body posture estimation data set, and preprocessing the data set.

[0089] In this embodiment, the electronic device may be a device with computing capabilities such as a laptop computer, a desktop computer, a workstation, a server, a field programmable gate array (FPGA), etc.

[0090] Wherein, the data set includes a training set and a test set, and data preprocessing is performed on them; the 3D human pose estimation data set used in this embodiment is the Human3.6M public data set. Specifically, determine or select a suitable human pose estimation data set. Check the integrity of the data set, remove or repair damaged files, outliers, or data with inconsistent labels. If the data set does not contain necessary pose annotations (such as joint point positions), manual annotation or automatic annotation tools are required. In order to increase the generalization ability of the model, the data set is enhanced, such as rotation, scaling, cropping, color conversion, etc. It can also include converting the data set into the format required for model training, including the processing of image files (such as conversion to a unified size and format) and the conversion of annotation files.

[0091] S102, obtaining a continuous two-dimensional posture sequence from the data set through a two-dimensional posture detector, and inputting the continuous two-dimensional posture sequence into STGFormer to obtain three-dimensional postures of continuous frames.

[0092] In this embodiment, first, a video containing human body motion is taken as input. A pre-trained 2D pose detector (such as OpenPose, DeepPose, etc.) is used to process each frame in the video to estimate a continuous 2D pose sequence. A spatiotemporal Graph Transformer (STGFormer) is constructed. The spatiotemporal Graph Transformer (STGFormer) is a deep learning model that combines the spatiotemporal graph (Spatial-Temporal Graph) and the Transformer architecture. The STGFormer is a 2D pose sequence with J joints and T frames. As input, the input channel size is 2. In order to enhance the model's ability to understand posture information, a linear layer (or fully connected layer) is used to map the original 2D posture coordinates from the two-dimensional space to the high-dimensional feature space. Specifically, P is linearly embedded. 2D Projection to high-dimensional feature vector The dimension of the feature vector representing each joint is d. In order to preserve the position information of each joint in space, spatial position embedding is introduced. These embeddings can be learnable position vectors, which are combined with the feature vector of each joint by addition or concatenation. The dimension of the position embedding should be the same as the dimension of the feature vector to ensure that the two can be seamlessly combined. Specifically, the spatial position embedding Add to the token, A 2D pose sequence containing joint position information is obtained. This step is used to preserve the joint position information in the spatial domain.

[0093] It should be noted that the token in the embodiment of the present application refers to a single gesture or a sequence.

[0094] After the spatial position is embedded, L STGFormer blocks are used to process the embedded features. Each STGFormer block combines graph convolution (for modeling the spatial relationship between joints) and self-attention mechanism (for modeling dynamic changes in time) to learn the spatiotemporal context information in the pose sequence. This method can fully capture the spatiotemporal characteristics of human motion. In order to obtain a 2D pose sequence containing a comprehensive representation of human motion.

[0095] Specifically, Figure 2 As shown in Figure 1, the STGFormer block is the core component of building the STGFormer model, which consists of two key components: spatial Graph Transformer and temporal Graph Transformer. These two components work together to fully capture the spatial and temporal features in human posture data.

[0096] In the spatial Graph Transformer, each body joint is treated as an independent token. This treatment allows the model to focus on learning the spatial relationship between body joints within each individual frame.

[0097] The implementation steps of the spatial Graph Transformer are:

[0098] Spatial Multi-head Self-Attention (spatial MSA) and Spatial Graph Convolutional Network (spatial GCN) are used to model the spatial correlation of all joints within a single frame.

[0099] Specifically, for the input spatial features Where B represents the batch size, and the calculation process of spatial GraphTransformer is shown in formula (1):

[0100]

[0101]

[0102] Among them, LN stands for layer normalization. GCN s Represents the spatial GCN, and the adjacency matrix is ​​represented by the topological matrix of the human skeleton. MSA s stands for Spatial MSA and MLP stands for Multilayer Perceptron.

[0103] It should be noted that in order to effectively model the spatial dependencies between joints, the spatial Graph Transformer uses a graph attention network or a multi-head self-attention mechanism (MSA) to aggregate global information. These mechanisms can dynamically adjust the attention weights between different joints to capture the complex interactions between joints. Although MSA is used to aggregate global information, the spatial Graph Transformer also combines a graph convolutional network (GCN) or similar graph neural network components to more finely capture local spatial relationships in key point sequences. GCN updates the representation of nodes by considering the neighbor information of the nodes, thereby enhancing the model's ability to understand local spatial structures.

[0104] Preferably, the processing process of the multi-head self-attention mechanism adopted by the spatial Graph Transformer includes:

[0105] Input Features Linear Mapping to Query key Sum Where n represents the sequence length and d represents the dimension of the input feature.

[0106] Calculate the token-to-token association matrix and transfer information between different tokens. The calculation formula of MSA is shown in formula (2):

[0107]

[0108] The query, key, and value are split H times and attention is performed in parallel. Then, the outputs of the H attention heads are concatenated.

[0109] Preferably, the implementation steps of the spatial graph convolutional network adopted by the spatial Graph Transformer are:

[0110] The GCN module is used to extract features. The calculation process is shown in formula (3):

[0111]

[0112] in, represents the adjacency matrix, express The sum of the elements along the diagonal, W1 and W2 represent the weight matrices, BN represents batch normalization, and σ represents the ReLU activation function.

[0113] In the Temporal Graph Transformer, each frame is treated as a separate token. This processing method allows the model to focus on learning the temporal correlation between frames, that is, the change pattern of human posture over time. In order to model the temporal dependency, the Temporal Graph Transformer also adopts the MSA mechanism. Through the self-attention mechanism, the model can capture the important temporal connections between different frames, thereby understanding the dynamic changes of the posture sequence.

[0114] The implementation of the Temporal Graph Transformer includes:

[0115] The spatial features output by the spatial Graph Transformer as input.

[0116] Will Transformed into From the time dimension, different motion trajectories of human joints are modeled through temporal multi-head self-attention (temporal MSA) and temporal graph convolutional network (temporal GCN). The modeling process is shown in formula (4):

[0117]

[0118] Where LN stands for layer normalization. T represents the time MSA, and the calculation process is shown in formula (2), which will not be repeated here. T represents temporal GCN, and the calculation process is shown in formula (3), which will not be repeated here. MLP stands for multi-layer perceptron. In temporal GCN, the similarity of a single joint between different time frames is calculated, and K nearest neighbors are selected as connecting nodes in the graph. Therefore, the adjacency matrix in temporal GCN is determined by the learned node features.

[0119] In order to reduce computational complexity and remove redundant information, a token pruning module is introduced. This module analyzes the importance of tokens and selects the most representative tokens for subsequent processing to obtain a 2D pose sequence containing tokens with high semantic diversity. Token pruning is usually performed by attention score calculation, threshold filtering, etc. The structure of the token pruning module can be found in Figure 3 .

[0120] Specifically, the implementation steps of the token pruning module are:

[0121] For input gesture tokens First, spatial redundancy is eliminated by averaging spatial pooling to obtain spatial pooling tokens

[0122] Apply K-Medoids clustering algorithm. For token Randomly select t tokens X j ∈X i As the initial cluster center, the distance between other tokens and the cluster center is then calculated and assigned to the nearest cluster. The calculation process is shown in formula (5):

[0123] d(X i , X j )=||X i -X j || 2 (5)

[0124] For each cluster center X j , calculate all tokens X in the cluster to which it belongs i The total distance to the cluster center is shown in formula (6):

[0125]

[0126] Among them, C j Represents the cluster center X j The cluster to which it belongs.

[0127] Select new cluster centers The total distance s from all tokens in the cluster to the cluster center j Minimum, as shown in formula (7):

[0128]

[0129] After the iteration is completed, the cluster center of each cluster is the final cluster center. The cluster center has high semantic diversity and contains more information than other tokens. Therefore, the cluster center is selected as the representative token. And other tokens are discarded to reduce redundant information in the video sequence.

[0130] After token pruning, a recovery head (which may be an upsampling layer or a decoder structure) is used to restore the pruned tokens to the original full-length sequence size. This step is to ensure that subsequent processing (such as 3D pose estimation) can be performed based on the complete pose sequence. Finally, one or more regression layers (or decoder structures) are used to estimate the 3D pose sequence from the restored full-length sequence. The output of this step is the 3D coordinates of each joint, thus enabling 3D pose estimation from 2D video input.

[0131] The recovery head consists of Multi-head Cross-Attention (MCA) and a linear regression head. MCA recovers the full-length sequence from a short-length sequence with high semantic spatiotemporal information. The linear regression head is used to estimate the 3D pose coordinates.

[0132] Specifically, the implementation steps of the Multi-head Cross-Attention (MCA) include:

[0133] The learnable tokens that the MCA will initialize to zero As Q, the representative token output by the last STGFormer block As K and V, and then perform residual connection, the calculation process of MCA is shown in formula (8):

[0134]

[0135] Among them, query key Sum d represents the dimension, {n q , n k , n v} are the number of tokens of {Q, K, V} respectively, is the recovered token, whose time dimension is the same as the learnable token same.

[0136] In summary, in order to solve the information redundancy and high computational cost problems of traditional Transformer-based models when processing posture sequences, the present application embodiment proposes the STGFormer architecture. This architecture adopts a unique "pruning first and then recovery" strategy, that is, only some key posture tokens are selected as the input of the STGFormer block during the processing process, and the full-length sequence is restored before the final output, thereby significantly improving the computational efficiency and reducing unnecessary computational overhead.

[0137] It should be noted that the embodiment of the present application uses the Human3.6M dataset to train STGFormer on an RTX 4090D, the batchsize is set to 16, the Adam optimizer is used to optimize the network parameters, the weight decay is 0.01, the initial learning rate is 0.0005, and the decay is 0.99 after each epoch.

[0138] The trained STGFormer model was tested on the Human3.6M dataset; subjects S1, S5, S6, S7, S8 in the dataset were used for training, and subjects S9 and S11 were used for testing. We adopted the most commonly used evaluation schemes: Scheme 1 (called P1) calculates the mean per-joint position error (MPJPE) between the estimated pose and the actual pose; Scheme 2 (called P2) calculates P-MPJPE, which is the MPJPE after aligning the predicted 3D pose with the actual pose using translation, rotation, and scaling.

[0139] In order to verify the effectiveness of STGFormer, the present application embodiment conducted a detailed comparative experiment on the Human3.6M 3D human pose estimation benchmark dataset. The experiment not only covers the direct performance competition with classic and advanced methods, but also deeply analyzes the impact of different video sampling frame numbers (T=27, 81, 243) on the balance between model complexity and performance.

[0140] Specifically, when using the 2D pose predicted by the pre-trained Stacked Hourglass model as input, STGFormer shows comparable performance to the state-of-the-art methods on both P1 and P2 error metrics. When using the real 2D pose as input, STGFormer's performance on P1 error is particularly outstanding. It is particularly noteworthy that in the lightweight configuration of T=27, STGFormer achieves a performance level comparable to MixSTE with only 4% of the floating point operations (FLOPs).

[0141] As the number of input video sampling frames increases, the performance of STGFormer steadily improves and the error continues to decrease. Under the fully sampled condition of T=243, STGFormer achieves excellent performance of P1 error of 41.4mm and P2 error of 34.3mm, while maintaining the lowest number of floating-point operations per frame, further demonstrating its advantage in computational efficiency. See Table (1) for details. Table (1) is a comparison of the method provided in the embodiment of the present application with other methods on the Human3.6M dataset.

[0142] Table (1)

[0143]

[0144] Where T is the number of input frames. CE is the 3D pose estimated only for the center frame. P1 is the MPJPE error (mm), and P2 is the P-MPJPE error (mm). represents the P1 error using the true 2D pose as input.

[0145] As shown in Table (1), compared with advanced methods such as StridedFormer, P-STMO, MHFormer, PoseFormerV2 and SlowFastFormer, which are also based on the Transformer architecture, STGFormer significantly reduces the model size and computational complexity while maintaining competitive performance, providing strong support for 3D human pose estimation applications in real-time or resource-constrained environments.

[0146] The second embodiment of the present invention provides a three-dimensional human body posture estimation device (such as Figure 4 ), which includes:

[0147] The linear embedding module 410 is used to map the original 2D posture coordinates from the two-dimensional space to the high-dimensional feature space. Specifically, P 2D Projection to high-dimensional feature vector The dimension of the feature vector represented by each joint is d;

[0148] A spatial position embedding module 420, for embedding the spatial position into the token to obtain a 2D pose sequence containing joint position information;

[0149] STGFormer module 430, including a spatial Graph Transformer and a temporal Graph Transformer, for capturing spatial and temporal features in human posture data;

[0150] a token pruning module 440 for selecting representative tokens for subsequent processing to obtain a 2D pose sequence containing high semantic diversity tokens;

[0151] A recovery header module 450 for restoring the pruned tokens to the original full-length sequence size;

[0152] The 3D pose estimation module 460 is used to estimate a 3D pose sequence from the restored full-length sequence.

[0153] A third embodiment of the present invention further provides an electronic device, the electronic device comprising:

[0154] at least one processor; and,

[0155] a memory communicatively connected to the at least one processor; wherein,

[0156] The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the three-dimensional human body posture estimation method of any of the aforementioned embodiments.

[0157] The fourth embodiment of the present invention further provides a non-transitory computer-readable storage medium, wherein the non-transitory computer-readable storage medium stores computer instructions, and the computer instructions are used to enable the computer to execute the three-dimensional human body posture estimation method described in any of the aforementioned embodiments.

[0158] The fifth embodiment of the present invention also provides a computer program product, which includes a computer program stored on a non-transitory computer-readable storage medium. The computer program includes program instructions. When the program instructions are executed by a computer, the computer executes the three-dimensional human body posture estimation method of any of the aforementioned embodiments.

[0159] The sixth embodiment of the present invention further provides a computer program, which includes program instructions. When the program instructions are executed by a computer, the computer is caused to execute the three-dimensional human body posture estimation method of any of the above embodiments.

[0160] Figure 5 A schematic diagram of a method or device 1000 that can implement an embodiment of the present invention is shown, which may include more or fewer devices than shown in the figure in some embodiments. In some embodiments, it can be implemented using a single or multiple devices. In some embodiments, it can be implemented using cloud or distributed devices.

[0161] like Figure 5 As shown, the device 1000 includes a processor 1001, which can perform various appropriate operations and processes according to the program and / or data stored in the read-only memory (ROM) 1002 or the program and / or data loaded from the storage part 1008 to the random access memory (RAM) 1003. Processor 1001 can be a multi-core processor or can include multiple processors. In some embodiments, processor 1001 can include a general main processor and one or more special coprocessors, such as a central processing unit (CPU), a graphics processing unit (GPU), a neural network processor (NPU), a digital signal processor (DSP), etc. In RAM 1003, various programs and data required for the operation of device 1000 are also stored. Processor 1001, ROM 1002 and RAM 1003 are connected to each other via bus 1004. Input / output (I / O) interface 1005 is also connected to bus 1004.

[0162] The processor and the memory are used together to execute the program stored in the memory. When the program is executed by the computer, the methods, steps or functions described in the above embodiments can be implemented.

[0163] The following components are connected to the I / O interface 1005: an input section 1006 including a keyboard, a mouse, a touch screen, etc.; an output section 1007 including a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker, etc.; a storage section 1008 including a hard disk, etc.; and a communication section 1009 including a network interface card such as a LAN card, a modem, etc. The communication section 1009 performs communication processing via a network such as the Internet. A drive 1010 is also connected to the I / O interface 1005 as needed. A removable medium 1011, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is installed on the drive 1010 as needed, so that a computer program read therefrom is installed into the storage section 1008 as needed. Figure 5 Only some components are schematically shown in the figure, which does not mean that the device 1000 only includes Figure 5 Components shown.

[0164] The systems, devices, modules or units described in the above embodiments may be implemented by a computer or its associated components. The computer may be, for example, a mobile terminal, a smart phone, a personal computer, a laptop computer, a vehicle-mounted human-computer interaction device, a personal digital assistant, a media player, a navigation device, a game console, a tablet computer, a wearable device, a smart TV, an Internet of Things system, a smart home, an industrial computer, a server or a combination thereof.

[0165] Although not shown, in an embodiment of the present invention, a computer-readable storage medium is provided, on which a computer program / instruction is stored. When the computer program / instruction is executed by a processor, the three-dimensional human body posture estimation method described in Example 1 is implemented.

[0166] Storage media in embodiments of the present invention include permanent and non-permanent, removable and non-removable items that can be used to store information by any method or technology. Examples of storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic tape magnetic disk storage or other magnetic storage devices or any other non-transmission medium that can be used to store information that can be accessed by a computing device.

[0167] Although not shown, an embodiment of the present invention further provides a computer program product, including: a computer program / instruction, which implements the three-dimensional human body posture estimation method described in Example 1 when executed by a processor.

[0168] The methods, programs, systems, devices, etc. of the embodiments of the present invention may be executed or implemented in a single or multiple networked computers, or may be practiced in a distributed computing environment. In the embodiments of this specification, in these distributed computing environments, tasks may be performed by remote processing devices connected via a communication network.

[0169] Those skilled in the art should understand that the embodiments of the present specification may be provided as methods, systems or computer program products. Therefore, those skilled in the art may imagine that the functional modules / units or controllers and related method steps described in the above embodiments may be implemented in software, hardware or a combination of software / hardware.

[0170] Unless explicitly stated, the actions or steps of the methods, programs, and embodiments of the present invention do not have to be performed in a specific order and can still achieve the desired results. In some implementations, multitasking and parallel processing are also possible or may be advantageous.

[0171] In this article, multiple embodiments of the present invention are described, but for the sake of brevity, the description of each embodiment is not exhaustive, and the same or similar features or parts between the embodiments may be omitted. In this article, "one embodiment", "some embodiments", "examples", "specific examples", or "some examples" are meant to be applicable to at least one embodiment or example according to the present invention, but not all embodiments. The above terms do not necessarily mean to refer to the same embodiment or example. In the absence of contradiction, those skilled in the art can combine and combine the different embodiments or examples described in this specification and the features of the different embodiments or examples.

[0172] The exemplary systems and methods of the present invention have been specifically shown and described with reference to the above embodiments, which are merely examples of the best modes for implementing the present systems and methods. It will be appreciated by those skilled in the art that various changes may be made to the embodiments of the systems and methods described herein when implementing the present systems and / or methods without departing from the spirit and scope of the present invention as defined in the appended claims.

Claims

1. A three-dimensional human body posture estimation method, characterized in that: include: Acquire a human body posture estimation data set, and preprocess the data set; Obtaining a continuous two-dimensional pose sequence from the data set through a two-dimensional pose detector, and inputting the continuous two-dimensional pose sequence into STGFormer to obtain three-dimensional poses of continuous frames; The three-dimensional poses of consecutive frames obtained by the STGFormer include: Using the continuous 2D pose sequence with J joints and T frames As input, where the input channel size is 2; By linear embedding P 2D Projection to high-dimensional features The feature dimension of each joint representation is d; Embedding spatial locations Add to the token to get a 2D pose sequence containing joint position information, where Using L STGFormer blocks to learn spatiotemporal context from the 2D pose sequence containing joint position information to obtain a 2D pose sequence containing a comprehensive motion representation of the human body; Selecting the 2D pose sequence containing the overall motion representation of the human body using a token pruning module to obtain a 2D pose sequence containing high semantic diversity tokens; The 2D pose sequence containing high semantic diversity tokens is restored to a full-length sequence through a recovery head, and a 3D pose sequence is estimated from the full-length sequence 2. The three-dimensional human body posture estimation method according to claim 1, characterized in that: The STGFormer block includes: a spatial Graph Transformer and a temporal Graph Transformer; The spatial Graph Transformer processes each body joint as a different token to learn the body joint relationship within each frame; The Temporal Graph Transformer treats each frame as a single token to learn the temporal correlations between frames.

3. The three-dimensional human body posture estimation method according to claim 2, characterized in that: The processing of the spatial Graph Transformer includes: The spatial correlation of all joints in a single frame is modeled using spatial multi-head self-attention and spatial graph convolutional networks, as follows: X′ s =X s +GCN s (LN(X s )) X″ s =X′ s +MSA s (LN(X′ s )) X″′ s =X″ s +MLP(LN(X″ s )) in, represents the spatial features of the input, LN represents layer normalization, GCN s Represents the spatial GCN, the adjacency matrix is ​​represented by the topological matrix of the human skeleton, MSA s stands for Spatial MSA and MLP stands for Multilayer Perceptron.

4. The three-dimensional human body posture estimation method according to claim 2, characterized in that: The temporal GraphTransformer processing includes: The spatial features output by the spatial Graph Transformer As input; Will Transformed into From the time dimension, different motion trajectories of human joints are modeled through temporal multi-head self-attention and temporal graph convolutional network. The modeling formula is as follows: X′ T =X T +GCN T (LN(X T )) X″ T =X′ T +MSA T (LN(X′ T )) X″′ T =X″ T +MLP(LN(X″ T )) Among them, LN stands for layer normalization, MSA T Representing time MSA, GCN T stands for Temporal GCN and MLP stands for Multi-Layer Perceptron.

5. The three-dimensional human body posture estimation method according to claim 1, characterized in that: The token pruning module processing includes: For input gesture tokens Spatial redundancy is eliminated by averaging spatial pooling to obtain spatial pooling tokens Randomly select t tokens X j ∈X i As the initial cluster center, The distances of other tokens from the cluster center are then calculated and assigned to the nearest cluster as follows: d(X i ,X j )=‖X i -X j || 2 , For each cluster center X j , calculate all tokens X in the cluster to which it belongs i Total distance to cluster center: Among them, C j Represents the cluster center X j The cluster to which it belongs; Select new cluster centers The total distance s from all tokens in the cluster to the cluster center j Minimum: After the iteration is completed, the cluster center of each cluster is the final cluster center, and the cluster center is selected as the representative token And other tokens are discarded to reduce redundant information in the video sequence.

6. The three-dimensional human body posture estimation method according to claim 1, characterized in that: The recovery head is composed of MCA and linear regression head; The processing steps of the MCA are: The learnable tokens that will be initialized to zero As Q, the representative token output by the last STGFormer block As K and V, and then perform residual connection as follows: Among them, query key Sum d represents the dimension, {n q , n k , n v } are the number of tokens of {Q, K, V} respectively, is the recovered token, whose time dimension is the same as the learnable token same; Finally, the 3D pose coordinates are estimated using the linear regression head 7. The three-dimensional human body posture estimation method according to any one of claims 3-4, characterized in that: The processing of the multi-head attention mechanism includes: input features Linear Mapping to Query key Sum Where n represents the sequence length and d represents the dimension of the input feature; Compute the token-to-token association matrix and pass information between different tokens as follows: Split the query, key, and value into H times and perform attention in parallel, and concatenate the outputs of the H attention heads; The processing of graph convolutional networks includes: extracting features, as follows: in, represents the adjacency matrix, express The sum of the elements along the diagonal, W1 and W2 represent the weight matrices, BN represents batch normalization, and σ represents the ReLU activation function.

8. A three-dimensional human body posture estimation device, characterized in that: include: A linear embedding module is used to map the original 2D pose coordinates from the two-dimensional space to the high-dimensional feature space; A spatial position embedding module is used to embed the spatial position into the token to obtain a 2D pose sequence containing joint position information; The STGFormer module includes a spatial Graph Transformer and a temporal Graph Transformer, which is used to capture spatial and temporal features in human posture data; A token pruning module to select representative tokens for subsequent processing to obtain a 2D pose sequence containing tokens with high semantic diversity; The recovery header module is used to restore the pruned tokens to the original full-length sequence size; The 3D pose estimation module is used to estimate the 3D pose sequence from the restored full-length sequence.

9. An electronic device, characterized in that: The electronic device includes: at least one processor; and, a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the three-dimensional human body posture estimation method described in any one of claims 1 to 7.

10. A non-transitory computer-readable storage medium, characterized in that: The non-transitory computer-readable storage medium stores computer instructions, which are used to enable the computer to perform the three-dimensional human posture estimation described in any one of claims 1 to 7.

Images