A three-dimensional human pose estimation method and device based on a double-stream encoder
By combining a dual-stream encoder with a Transformer and a dynamic graph convolutional network, the problems of insufficient local structure modeling and unstable training of deep networks in 3D human pose estimation are solved, achieving more efficient 3D human pose estimation.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- ZHEJIANG UNIV OF TECH
- Filing Date
- 2026-03-16
- Publication Date
- 2026-07-14
AI Technical Summary
Existing 3D human pose estimation methods suffer from problems such as insufficient local structure modeling, difficulty in adapting to dynamic joint relationships, and instability in deep network training, which limit the upper limit of model performance.
A method based on a two-stream encoder is adopted, which combines the global modeling capability of Transformer and the local dynamic modeling capability of dynamic graph convolutional network with a stacked two-stream encoder. An adaptive fusion module and a two-layer normalization strategy are introduced to solve the gradient instability problem.
It significantly improves the robustness and accuracy of the model under complex motions, enhances the model's convergence speed and prediction accuracy, and is better able to adapt to dynamic joint relationships and complex motion sequences.
Smart Images
Figure CN122391769A_ABST
Abstract
Description
Technical Field
[0001] This application belongs to the field of computer vision and deep learning technology, specifically relating to a three-dimensional human pose estimation method and apparatus based on a two-stream encoder. Background Technology
[0002] Monocular 3D human pose estimation (HPE) aims to predict the 3D position of human joints from a single-view image or video. Current mainstream methods employ a two-stage process: first, 2D keypoints are detected, and then the 3D spatial position is determined using a 2D-to-3D lifting network.
[0003] Existing methods are mainly divided into Transformer-based methods and Graph Convolutional Network (GCN)-based methods. While Transformer-based methods can effectively model long sequence dependencies, they are typically computationally complex and have weak local structure awareness. Conversely, GCN-based methods utilize the human skeletal structure for modeling, resulting in high computational efficiency. However, they are limited by a fixed adjacency matrix, making it difficult to adapt to dynamically changing joint relationships, and they also have shortcomings in capturing long-distance dependencies.
[0004] Although hybrid methods have been developed to combine the two, they typically rely on static graph structures and cannot effectively integrate dynamic spatial dependencies, thus limiting their ability to model complex motion sequences.
[0005] In addition, gradient instability, a common problem in deep network training, also limits the upper limit of model performance. Summary of the Invention
[0006] The purpose of this application is to provide a three-dimensional human pose estimation method and device based on a two-stream encoder, so as to solve the problems of insufficient local structure modeling, difficulty in adapting to dynamic joint relationships, and instability of deep network training in the existing technology.
[0007] To achieve the above objectives, the technical solution of this application is as follows:
[0008] A three-dimensional human pose estimation method based on a two-stream encoder includes:
[0009] Two-dimensional skeleton data of the target human body is obtained, projected onto the latent feature space through a linear layer, and positional encoding is performed to obtain skeleton features;
[0010] The skeleton features are input into a two-stream encoder that is stacked in two-stream blocks. In each two-stream block, the input features are passed through a transformer branch and a dynamic graph convolution branch, respectively. Global features are extracted through the transformer branch and local features are extracted through the dynamic graph convolution branch.
[0011] In each two-stream block, global and local features are input into an adaptive fusion module to obtain fused features;
[0012] The fused features output from the dual-stream encoder are input into the regression head, which outputs the final predicted 3D human pose.
[0013] Preferably, the converter branch performs the following operations on the input features:
[0014] The input features are linearly transformed through three learnable linear projection matrices to obtain the query vector, key vector, and value vector.
[0015] The query vector, key vector, and value vector are divided into several heads along the feature dimension;
[0016] Normalize the query vector, key vector, and each head of the key vector separately, and perform attention operations to obtain the attention features of each head;
[0017] The attention features of all heads are concatenated and then linearly transformed to obtain the global features of the converter branch output.
[0018] Preferably, the dynamic graph convolution branch performs the following operations:
[0019] The relationship between joint feature vectors is calculated using a learnable compatibility function to obtain the local adjacency matrix;
[0020] The global affinity matrix is obtained by projecting the input features onto a learnable projection matrix.
[0021] The local adjacency matrix and the global affinity matrix are weighted and fused to generate a spatial dual adjacency matrix;
[0022] The dual adjacency matrix is used to perform graph convolution operation on the input features to obtain spatially updated skeleton features;
[0023] A temporal adjacency matrix is constructed based on feature similarity, and temporal feature aggregation is performed to obtain the output features of the dynamic graph spacing branch.
[0024] Preferably, in each dual-stream block, global and local features are input to an adaptive fusion module to obtain fused features, including:
[0025] Global and local features are concatenated along the channel dimension to obtain combined features;
[0026] The combined features are linearly mapped to obtain the weights corresponding to the global and local features. Then, the global and local features are weighted and fused to obtain the fused features.
[0027] Preferably, the execution time-series feature aggregation is performed using the following aggregation formula:
[0028]
[0029] in, This represents the features obtained by aggregating time-series features. For activation function, For normalization operations, Represents a set of neighbors. Indicates the first Frame and the Frames in the temporal adjacency matrix The corresponding value in the matrix represents the temporal adjacency matrix. Indicates weight, , Indicates the joint in the first position Frame and the The feature vector of a frame.
[0030] This application also proposes a three-dimensional human pose estimation device based on a dual-stream encoder, including a processor and a memory storing a number of computer instructions, which, when executed by the processor, implement the steps of the above method.
[0031] This application provides a three-dimensional human pose estimation method and apparatus based on a two-stream encoder, which has the following advantages compared with the prior art:
[0032] 1. A novel dual-stream hybrid architecture is proposed, which effectively combines the global modeling capability of Transformer with the local dynamic modeling capability of DGCN.
[0033] 2. A dynamic graph convolution module was designed, which significantly improves the robustness of the model under complex actions by fusing local and global graph views.
[0034] 3. A two-layer normalization strategy was introduced to solve the gradient instability problem in deep Transformer networks, thereby improving the convergence speed and accuracy of the model. Attached Figure Description
[0035] Figure 1 This is a flowchart of the 3D human pose estimation method based on two-stream Transformer and dynamic graph convolution in this application. Detailed Implementation
[0036] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.
[0037] One embodiment of this application, such as Figure 1As shown, a three-dimensional human pose estimation method based on a two-stream encoder is provided, including:
[0038] Step S1: Obtain the two-dimensional skeleton data of the target human body, project it onto the latent feature space through a linear layer, and perform position encoding to obtain skeleton features.
[0039] For a target human body to be used for 3D human pose estimation, its 2D skeleton data is first obtained. , Where T is the number of frames, J is the number of joints, and 3 represents the feature dimension of each joint, which typically includes the horizontal coordinate x, the vertical coordinate y, and the confidence score c of that joint.
[0040] Next, in order to convert the low-dimensional geometric coordinate information into high-dimensional semantic features for processing by deep neural networks, the two-dimensional skeleton data is... The input is fed into a linear layer containing a learnable weight matrix and bias terms. This linear layer reduces the feature dimension of the input data from the initial... Mapping to a higher latent feature dimension d makes the data form change from Transform into This process effectively increases the channel width of the features, thereby enabling each joint node to have a richer feature representation capability in the latent space. Considering the permutation invariance of the subsequent processing network, in order for the network to explicitly distinguish different human joint identities (e.g., distinguishing between the left wrist and the right ankle), spatial location encoding is needed to obtain skeletal features, which serve as input features for the subsequent network.
[0041] Step S2: Input the skeleton features into a two-stream encoder with stacked two-stream blocks. In each two-stream block, the input features pass through a converter branch and a dynamic graph convolution branch, respectively. Global features are extracted through the converter branch, and local features are extracted through the dynamic graph convolution branch.
[0042] This embodiment of the dual-stream encoder includes multiple stacked dual-stream blocks, each containing a parallel Transformer branch and a Dynamic Graph Convolutional Branch (DGCN). Within each dual-stream block, the Transformer branch and the DGCN branch receive the same input feature from the previous dual-stream block in parallel. Its dimensions are ,in For batches, For frame number, For the number of joints, For feature dimensions. In the converter branch, the number of joints... Treating the input features as a sequence of length L Remodeling The input features are presented in the form of query, key, and value vector transformations. In the dynamic graph convolution branch, the input features are... Remodeling The number of joints is in the form of Number of nodes in a graph Feature Dimension Number of feature channels as a node .
[0043] The transformer branch is primarily used to capture long-range spatiotemporal dependencies. Addressing the gradient instability issue caused by the unbounded dot product in the standard Transformer, this embodiment independently applies a two-layer normalization strategy to the vector of each head in the multi-head attention mechanism. This means applying layer normalization independently to two information flows: first, the interaction flow, which normalizes the query vector Q and key vector K, using the normalized dot product to constrain the attention score and prevent the Softmax function from entering the saturation region, thus preventing gradient vanishing; second, the content flow, which normalizes the value vector V to maintain the consistency of feature distribution and preserve nonlinear expressiveness.
[0044] Specifically, in the converter branch, the input feature processing flow is as follows:
[0045] Step 2.1.1: Perform linear transformations on the input features using three learnable linear projection matrices to obtain the query vector, key vector, and value vector.
[0046] This step will input features. Each is achieved through three learnable linear projection matrices. , Perform a linear transformation to obtain the query vector Q, the key vector K, and the value vector V.
[0047] Step 2.1.2: Divide the query vector, key vector, and value vector into several heads along the feature dimension.
[0048] To capture information from different subspaces, the query vector Q, key vector K, and value vector V are divided into h heads along the feature dimension d. For the i-th head (where i = 1, ..., h), its corresponding query, key, and value vectors are denoted as follows: ,in The feature dimensions for each head.
[0049] Step 2.1.3: Normalize each head of the query vector, key vector, and key vector respectively, and perform attention operation to obtain the attention features of each head.
[0050] The normalization process is decoupled into an interaction flow and a content flow: in the interaction flow, the query vector of the i-th head is processed separately. and key vector Normalization yields and Its technical function is to enforce a numerical range constraint before the dot product calculation, preventing the Softmax input from falling into the saturation region and thus ensuring effective gradient propagation; while in the content stream, it independently constrains the value vector. Normalization yields The aim is to maintain the stability of semantic feature distribution while avoiding interference from the strength of interaction flow constraints, and to preserve the nonlinear expressive power of features to the maximum extent.
[0051] Then, based on the normalized vector, the attention features for each head i are calculated:
[0052]
[0053] Norm represents normalization.
[0054] Step 2.1.4: Concatenate the attention features of all heads and then perform a linear transformation to obtain the global features of the converter branch output.
[0055] Output features of all h heads Concatenate the features along the channel dimension to obtain the concatenated features. Then, use a learnable linear output matrix. A linear transformation is performed on the concatenated features, and the information from each head is fused to obtain the global features output by the converter branch. :
[0056] .
[0057] It's important to note that standard Transformers typically normalize the overall features before or after attention computation. In deep networks, the dot product of the query vector and key vector can be unbounded, leading to excessively large values. This causes the Softmax function to enter the saturation region (i.e., the output approaches 0 or 1), resulting in minimal gradients and making it difficult for deep networks (e.g., those exceeding 20 layers) to train and converge. This application employs a dual-layer normalization strategy to prevent Softmax saturation, ensuring effective gradient propagation and significantly improving the training stability of deep networks. The dual-layer normalization strategy also preserves the non-linear expressiveness of the feature content, avoiding the loss of detailed information due to excessive smoothing. This allows the model in this application to easily scale to 40 layers or even deeper, thereby capturing richer spatiotemporal features.
[0058] The Dynamic Graph Spacing Branch (DGCN) aims to utilize input features to construct spatial adjacency relationships that can simultaneously capture both local skeletal structures and global dependencies in the human skeleton.
[0059] In the Dynamic Graph Spacing Branch (DGCN), the processing flow of input features is as follows:
[0060] Step 2.2.1: Calculate the relationship values between joint feature vectors using a learnable compatibility function to obtain the local adjacency matrix.
[0061] Utilize a learnable compatibility function Calculate the short-range relationships between nodes and construct a local adjacency matrix. The calculation formula is:
[0062]
[0063] in They represent the first The and the first Feature vectors of each joint.
[0064] Step 2.2.2: Project the input features using a learnable projection matrix to obtain the global affinity matrix.
[0065] Using learnable projection matrices Projecting the input features, inferring long-range dependencies between nodes, and constructing a global affinity matrix. The calculation formula is:
[0066]
[0067] Step 2.2.3: Perform weighted fusion of the local adjacency matrix and the global affinity matrix to generate the spatial dual adjacency matrix.
[0068] To integrate local and global information, a learnable adaptive fusion coefficient is introduced. The two matrices are then weighted and fused to generate the final spatial dual adjacency matrix. The calculation formula is as follows:
[0069]
[0070] In addition, during training, the local geometric adjacency matrix is calculated. With global affinity matrix The difference between them is treated as a structural consistency loss, which forces both to maintain structural consistency by minimizing the loss, thereby stabilizing the learning of dynamic topologies.
[0071] Step 2.2.4: Perform graph convolution operation on the input features using the dual adjacency matrix to obtain the spatially updated skeleton features.
[0072] Based on the generated dual adjacency matrix For input features Performing a graph convolution operation, which can be represented as:
[0073]
[0074] in, It is a degree matrix. , For weight parameters, For activation function, For normalization.
[0075] Step 2.2.5: Construct a temporal adjacency matrix based on feature similarity and perform temporal feature aggregation to obtain the output features of the dynamic graph spacing branch.
[0076] After updating the skeleton features in the spatial dimension in the previous step, in order to capture the motion dependence of the same joint at different times, this step uses the K-nearest neighbor algorithm to adaptively construct temporal connections and aggregates the features of related frames through graph convolution operations.
[0077] For each joint node in the human skeleton, calculate its position at any two different time frames in the entire video sequence (denoted as ). and The feature similarity between () is used in this embodiment. The dot product is used as the similarity measure, and the calculation formula is:
[0078]
[0079] in and These represent the joint at the [number]th [year]. Frame and the The feature vector of a frame.
[0080] Then, a temporal adjacency matrix (temporal dynamic graph) is constructed. Based on the similarity calculated above, the current frame is used as the reference. The K frames with the highest similarity scores in the entire sequence (i.e., the K nearest neighbors) are denoted as the neighbor set. Based on this, an adjacency matrix is constructed along the time dimension. If frame belong Then at the corresponding position in the matrix Set to 1 otherwise to 0. This process adaptively determines which frames are "relevant" based on the actual motion conditions.
[0081] Finally, temporal feature aggregation (message passing) is performed based on the constructed temporal adjacency matrix. Using a learnable time weight matrix The features of all frames are weighted and aggregated. Specifically, for the current frame... , and its time neighbor frames eigenvectors After linear transformation, the features are accumulated according to the connection relationships defined by the adjacency matrix, and then residually connected with the features of the current frame itself. The aggregation formula is as follows:
[0082]
[0083] in, For activation function, This is a normalization operation. The output features obtained through this aggregation operation are... It not only includes information from the current moment, but also incorporates auxiliary information from the frames in the entire video sequence that are most similar to its motion state, thereby effectively enhancing the robustness of the features.
[0084] This step, in the temporal dimension, aims to capture motion dependencies across frames. By performing message passing operations on the constructed temporal graph, motion information from relevant past and future frames is aggregated to obtain the final output features of the DGCN branch.
[0085] Step S3: In each dual-stream block, global and local features are input into the adaptive fusion module to obtain fused features.
[0086] In each dual-stream block of the dual-stream encoder China (among them) This requires effectively integrating the global features extracted by the transformer branch with the local dynamic features extracted by the dynamic graph convolution branch. The specific processing flow is as follows:
[0087] Step 3.1: Concatenate the global features and local features along the channel dimension to obtain the combined features.
[0088] Obtain the global characteristics of the converter branch output in the current dual-stream block. Local features of the output of convolution branches in dynamic graphs Concat the data along the channel dimension to obtain combined features.
[0089] Step 3.2: Perform linear mapping on the combined features to obtain the weights corresponding to the global and local features. Then, perform weighted fusion on the global and local features to obtain the fused features.
[0090] In this embodiment, the combined features are input into a learnable linear layer for mapping, with the output dimension of this linear layer configured to be 2. Through this mapping, the high-dimensional combined features are compressed into a vector containing two values, representing the network's original scores for the importance of the two branches. Subsequently, this vector is normalized using a Softmax activation function, resulting in two scalar values that sum to 1. These two scalar values are then assigned sequentially as transformer flow weights. and dynamic graph flow weights The process is as follows:
[0091]
[0092] This process enables the network to determine which branch's features are more important based on the current motion state of the input.
[0093] Finally, using the calculated weights, the features of the two branches are linearly weighted and summed to obtain the final output features of the current two-stream block. The calculation formula is as follows:
[0094]
[0095] This output feature This will be used as the input to the next two-stream block, or (if it's the last layer) as the input to the regression head. In this embodiment, a "two-stream block" refers to a complete processing unit containing the two parallel branches and the adaptive fusion module. To capture both spatial and temporal information simultaneously, each two-stream block employs a "space-first, time-later" concatenated processing architecture. Specifically, for the converter branch, spatial transformation is performed first, treating the input features as a spatial sequence, and then performing linear projection, two-layer normalization, and spatial attention to capture the dependencies between different joints within the same frame. This is followed by temporal transformation, reshaping the spatially transformed features into a temporal sequence, and then performing the aforementioned operations again to capture the temporal interactions between different frames, ultimately obtaining the converter branch features. Similarly, for the dynamic graph convolution branch, spatial dynamic graph convolution is performed first, generating a spatial dual adjacency matrix through adaptive fusion and then performing graph convolution to update the skeleton topological features; subsequently, temporal dynamic graph convolution is performed, constructing a temporal dynamic graph based on feature similarity and performing temporal message passing, ultimately obtaining the dynamic graph convolution branch features. This means that before the input features enter the adaptive fusion module, they have already independently completed the deep feature extraction in the spatial and temporal domains within their respective branches.
[0096] Step S4: Input the fused features output by the dual-stream encoder into the regression head to output the final predicted 3D human pose.
[0097] After N layers of stacked two-stream blocks, the features output by the Nth two-stream block are still high-dimensional latent features, not the final 3D pose coordinates. To obtain the 3D pose, these features need to be input into a regression head composed of fully connected layers, which projects the feature map onto the 3D motion space, thereby outputting the final predicted 3D human pose sequence.
[0098] To verify the effectiveness of the technical solution of this application, Table 1 shows the quantitative comparison results of the method of this application with current mainstream advanced methods (including MotionBERT, MotionAGFormer, etc.) and variants of the method of this application (as shown in Table 2) on the Human3.6M dataset. In the horizontal column of the table, "T" represents the number of frames of the input video sequence, "MPJPE" represents the average joint position error (unit: mm), and the middle columns (such as Dire, Sit, Walk, etc.) correspond to 15 specific action categories in the dataset (Dire for direction, Disc for discussion, Eat for eating, Greet for greeting, Phone for making a call, Photo for taking a photo, Pose for posing, Purchase for shopping, Sit for sitting, SitD for sitting down, Smoke for smoking, Wait for waiting, WalkD for walking the dog, Walk for walking, WalkT for walking in groups), which aims to evaluate the fine-grained performance of the model under different motion amplitudes and occlusion conditions. "Avg" is the average error value of all action categories; the vertical column lists the various existing models participating in the comparison and the different configuration variants proposed in this application.
[0099] Table 1
[0100]
[0101] Table 2
[0102]
[0103] As shown in Table 1, the proposed method (especially the DualSTFormer-L variant) achieves the best performance. Under the MPJPE metric, the average error of the proposed method is only 37.6 mm, and under the P-MPJPE metric, the average error is 31.5 mm. Compared with the high-performing MotionAGFormer in recent years, the proposed method reduces the MPJPE error by 0.8 mm (from 38.4 mm to 37.6 mm) and by 1.6 mm compared with MotionBERT. This indicates that the proposed dual-stream architecture can more effectively utilize spatiotemporal features and significantly improve the accuracy of attitude estimation.
[0104] As can be seen from the results of each specific action category in Table 1, the method of this application has a particularly significant advantage in challenging dynamic actions. For example, in actions such as "Directions" and "Sitting," human joints exhibit complex self-occlusion and rapid changes. The dynamic graph structure constructed by the method of this application using DGCN streams can adaptively adjust the topological connections according to input features, thereby maintaining low error in these complex scenarios. Compared with the GLA-GCN method that relies on static graph structures, the method of this application achieves a performance improvement of approximately 15% in average error, fully verifying the necessity of combining dynamic graph convolution with the global Transformer attention mechanism. In addition, as shown in Table 2, in order to adapt to different computational resource constraints and meet diverse application needs, the embodiments of this application design four model variants with different configurations, namely DualSTFormer-XS, -S, -B, and -L. These variants adjust the number of network layers N, channel dimension, etc. Different performance levels are achieved by varying the number of input frames T. Even the lightweight XS version achieves an MPJPE of 44.4 mm, maintaining strong competitiveness while keeping computationally low. Prediction accuracy steadily improves with increasing model size (from XS to L), demonstrating the good scalability of the proposed architecture.
[0105] To further verify the generalization performance of the proposed method in complex backgrounds and different environments, this application was tested on the MPI-INF-3DHP dataset. This dataset includes various scenes such as green screen backgrounds, non-green screen indoor backgrounds, and complex outdoor backgrounds, and the types of motion are more diverse, which places higher demands on the robustness of the model. Specifically, Table 3 lists the various algorithm models involved in the comparison in the vertical column, including existing technologies such as PoseFormer and GLA-GCN, as well as different parameter scale variants of DualSTFormer proposed in this application; in the horizontal column, "T" represents the number of input video frames processed by the model, "MPJPE" (Mean Per Joint Position Error) represents the average joint position error, the smaller the value, the higher the prediction accuracy, "PCK" (Percentage of Correct Keypoints) represents the keypoint accuracy rate at a threshold of 150mm, that is, the proportion of predicted points and ground truth points with a distance of less than 150mm, the higher the value, the more accurate the localization, and "AUC" (Area Under the Curve) represents the area under the PCK curve, which can more comprehensively reflect the overall performance of the model under different error tolerances; by comparing these indicators, it is possible to intuitively verify whether the model has stronger generalization ability and structural robustness when facing complex uncontrolled scenes such as outdoor and green screen.
[0106] Table 3
[0107]
[0108] As shown in Table 3, the DualSTFormer-L model proposed in this application sets a new state-of-the-art record on this dataset. Specifically, the proposed method achieves an MPJPE of 14.0 mm, a PCK accuracy of 99.3%, and an AUC of 90.6. Compared to the current state-of-the-art baseline method, MotionAGFormer-L (MPJPE of 16.2 mm), the proposed method reduces the prediction error by 13.6% (from 16.2 mm to 14.0 mm). Simultaneously, the proposed method achieves an AUC of 90.6, significantly higher than MotionAGFormer's 85.3, reflecting the overall prediction quality. This fully demonstrates the substantial progress in accuracy achieved by the proposed model. MPI-INF-3DHP contains a large number of outdoor scenes that the model had not seen during training. Traditional static graph convolutional (GCN) methods rely on fixed skeletal connections, making it difficult to adapt to such environmental changes and unknown complex poses.
[0109] The superior results achieved by the method in this application are primarily attributed to the dynamic graph construction module in the DGCN stream. This module can infer the affinity relationships between nodes in real time based on the input features, adaptively constructing local and global topologies. Experimental data shows that even in complex outdoor scenes, the adaptive fusion mechanism of this application can effectively filter noise interference and maintain structural consistency, thus achieving an extremely high accuracy of 99.3% on the PCK metric. Furthermore, Table 3 shows that the lightweight version of this application (DualSTFormer-XS, T=27 frames) still achieved an MPJPE of 18.3mm with a very small number of input frames, which is superior to many existing methods with long input sequences (such as STCFormer's 23.1mm). This indicates that the method in this application can efficiently extract features and has the potential for real-time 3D pose estimation on computationally limited devices.
[0110] Another embodiment of this application provides a three-dimensional human pose estimation device based on a dual-stream encoder, including a processor and a memory storing a plurality of computer instructions, which, when executed by the processor, implement the steps of the above method.
[0111] Specific limitations regarding the 3D human pose estimation device based on a two-stream encoder can be found in the limitations of the 3D human pose estimation method based on a two-stream encoder mentioned above, and will not be repeated here. The aforementioned 3D human pose estimation device based on a two-stream encoder can be implemented entirely or partially through software, hardware, or a combination thereof. It can be embedded in or independent of the processor in a computer device in hardware form, or stored in the memory of a computer device in software form, so that the processor can call and execute the above operations.
[0112] The memory and processor are electrically connected directly or indirectly to enable data transmission or interaction. For example, these components can be electrically connected to each other via one or more communication buses or signal lines. The memory stores a computer program that can run on the processor, which implements the three-dimensional human pose estimation method based on a dual-stream encoder in this embodiment of the invention by running the computer program stored in the memory.
[0113] The memory may be, but is not limited to, Random Access Memory (RAM), Read Only Memory (ROM), Programmable Read-Only Memory (PROM), Erasable Programmable Read-Only Memory (EPROM), Electrically Erasable Programmable Read-Only Memory (EEPROM), etc. The memory stores the program, and the processor executes the program upon receiving an execution instruction.
[0114] The processor may be an integrated circuit chip with data processing capabilities. The aforementioned processor can be a general-purpose processor, including a Central Processing Unit (CPU), a Network Processor (NP), etc. It can implement or execute the methods, steps, and logic block diagrams disclosed in the embodiments of this invention. The general-purpose processor can be a microprocessor or any conventional processor.
[0115] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the invention patent. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this patent application should be determined by the appended claims.
Claims
1. A three-dimensional human pose estimation method based on a two-stream encoder, characterized in that, The three-dimensional human pose estimation method based on a two-stream encoder includes: Two-dimensional skeleton data of the target human body is obtained, projected onto the latent feature space through a linear layer, and positional encoding is performed to obtain skeleton features; The skeleton features are input into a two-stream encoder that is stacked in two-stream blocks. In each two-stream block, the input features are passed through a transformer branch and a dynamic graph convolution branch, respectively. Global features are extracted through the transformer branch and local features are extracted through the dynamic graph convolution branch. In each two-stream block, global and local features are input into an adaptive fusion module to obtain fused features; The fused features output from the dual-stream encoder are input into the regression head, which outputs the final predicted 3D human pose.
2. The three-dimensional human pose estimation method based on a two-stream encoder according to claim 1, characterized in that, The converter branch performs the following operations on the input features: The input features are linearly transformed through three learnable linear projection matrices to obtain the query vector, key vector, and value vector. The query vector, key vector, and value vector are divided into several heads along the feature dimension; Normalize the query vector, key vector, and each head of the key vector separately, and perform attention operations to obtain the attention features of each head; The attention features of all heads are concatenated and then linearly transformed to obtain the global features of the converter branch output.
3. The three-dimensional human pose estimation method based on a two-stream encoder according to claim 1, characterized in that, The dynamic graph convolution branch performs the following operations: The relationship between joint feature vectors is calculated using a learnable compatibility function to obtain the local adjacency matrix; The global affinity matrix is obtained by projecting the input features onto a learnable projection matrix. The local adjacency matrix and the global affinity matrix are weighted and fused to generate a spatial dual adjacency matrix; The dual adjacency matrix is used to perform graph convolution operation on the input features to obtain spatially updated skeleton features; A temporal adjacency matrix is constructed based on feature similarity, and temporal feature aggregation is performed to obtain the output features of the dynamic graph spacing branch.
4. The three-dimensional human pose estimation method based on a two-stream encoder according to claim 1, characterized in that, In each dual-stream block, global and local features are input to an adaptive fusion module to obtain fused features, including: Global and local features are concatenated along the channel dimension to obtain combined features; The combined features are linearly mapped to obtain the weights corresponding to the global and local features. Then, the global and local features are weighted and fused to obtain the fused features.
5. The three-dimensional human pose estimation method based on a two-stream encoder according to claim 3, characterized in that, The execution time sequence feature aggregation formula is as follows: in, This represents the features obtained by aggregating time-series features. For activation function, For normalization operations, Represents a set of neighbors. Indicates the first Frame and the Frames in the temporal adjacency matrix The corresponding value in the matrix represents the temporal adjacency matrix. Indicates weight, , Indicates the joint in the first position Frame and the The feature vector of a frame.
6. A three-dimensional human pose estimation device based on a dual-stream encoder, comprising a processor and a memory storing a plurality of computer instructions, characterized in that, When the computer instructions are executed by the processor, they implement the steps of the method according to any one of claims 1 to 5.