A 3D human pose estimation method integrating local and global features

Through the parallel fusion network, the global and local features of 3D human poses were captured and fused separately, which solved the problem of insufficient feature utilization in the existing model, and significantly improved the generalization performance of the 3D human pose estimation model.

CN114565938BActive Publication Date: 2024-11-26SOUTHEAST UNIV +1
View PDF 0 Cites 2 Cited by

Patent Information

Application Number
CN202210142459.X
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2022-02-16
Publication Date
2024-11-26
Estimated Expiration
2042-02-16

AI Technical Summary

Technical Problem

The existing 3D human pose estimation model fails to effectively discover and integrate global and local features in human poses, resulting in insufficient generalization performance.

Method used

A parallel fusion network is designed to capture the global and local features of human poses through a fully connected network and a packet-connected network, and fuse them to improve the generalization performance of the 3D human pose estimation model.

Benefits of technology

Through the proposed method, the generalization performance of the 3D human pose estimation model is significantly improved. The experimental results show that the average improvement of advanced model methods such as fully connected neural network, graph neural network and local connected network is 26.1%, and it is widely used across data sets Best performance in terms of chemical ability.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN114565938B_ABST
    Figure CN114565938B_ABST
Patent Text Reader

Abstract

The present invention provides a method for 3D human posture estimation that integrates local and global features, including: proposing an observation mode I that can be applied to a 3D human posture estimation model; proposing an observation mode II that can be applied to a 3D human posture estimation model; adjusting a trained image-2D posture network; a fully connected module that captures the global features of human posture; a grouped connection module that captures the local features of human posture; the fusion of local features and global features is regressed to 3D posture, and finally a model for 3D human posture estimation is obtained. The beneficial effects of the present invention are: proposing a method for 3D human posture estimation that integrates local and global features, by respectively using a fully connected module and a grouped connection module to capture the global features and local features of human posture, and then fusing the extracted features, a 3D human posture estimation model with better generalization performance is obtained, and the posture estimation performance index with high local feature similarity is significantly improved.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to a three-dimensional human body posture estimation method integrating local and global features, and belongs to the technical field of computer vision based on deep learning. Background Art

[0002] Human pose estimation is a long-standing computer vision problem that aims to construct a representation of the human body (such as human skeleton and body shape) from specific input content (images or videos) with human poses. The representation of human poses in three-dimensional space is called 3D human pose estimation. 3D human pose estimation has a wide range of applications, such as in human-computer interaction, virtual reality (VR), motion analysis and other computer vision application fields. The development of deep learning technology, with the powerful representation learning ability of deep network models, has greatly improved the performance of 3D human pose estimation model methods, and made human pose estimation methods based on deep learning a popular solution at this stage. 3D human pose estimation algorithms based on deep learning can be mainly divided into two categories: one-stage 3D human pose estimation algorithms and two-stage 3D human pose estimation algorithms. One-stage 3D human pose estimation algorithms directly establish a regression mapping relationship from images to 3D human poses. Due to the inherent depth ambiguity from a single image to 3D human poses, one-stage 3D human pose estimation algorithms do not perform well. The two-stage 3D human pose estimation algorithm first regresses the 2D human key point positions from the image, and then transforms the 2D human key point mapping to the 3D space. Thanks to the remarkable results of the 2D human pose estimation model, the two-stage 3D human pose estimation algorithm has gradually become the mainstream algorithm for 3D human pose estimation. Although the human pose estimation method based on deep learning has achieved great results, for the specific task of 3D human pose estimation, how to explore and utilize the unique structural information of the human body as much as possible has become a key issue. At the same time, improving the generalization performance of the 3D human pose estimation model based on deep learning is still challenging. Surprisingly, we found some encouraging observation patterns in the existing 3D human pose estimators: the prediction error of the human key points is highly correlated with the structure of the human pose, and different human pose cross-actions have high similarity in local features. These observation patterns indicate that different human poses have common global features and regular similar local features. In order to realize the idea of ​​designing the network based on the observation of global features and local features, a parallel fusion network is proposed, in which the fully connected network and the grouped connection network capture the global features and local features of the human pose respectively. The fully connected network connects all input features and output features indiscriminately to learn global information. The group-connected network connects input features and output features in groups using global information representation, focusing on learning local feature information of human pose. Based on this motivation, we design a parallel fusion network that fuses global and local features of human structure information for 3D human pose estimation. Summary of the invention

[0003] Technical issues:

[0004] In view of the fact that the above-mentioned 3D human posture estimation model does not effectively explore and integrate the global features and local features in the human posture, the purpose of the present invention is to provide a three-dimensional human posture estimation method that integrates local and global features. Based on the regular observation patterns observed in human posture, a parallel fusion network is proposed, in which a fully connected network and a grouped connection network capture the global features and local features of the human posture respectively, ultimately improving the generalization performance of the 3D human posture estimation model.

[0005] Technical solution:

[0006] To achieve the above object, the present invention adopts the following technical scheme: a 3D human body posture estimation method integrating local and global features, comprising:

[0007] Step 1: Adjust the image-2D posture network completed by training the cascade pyramid network, and use the experimental data set to adjust the training on the trained model to obtain the first stage model of the 3D human posture estimation model;

[0008] Step 2: Propose an observation mode I that can be applied to the 3D human posture estimation model to characterize the correlation between the error of key points and human posture, and guide the design of a fully connected module;

[0009] Step 3: Propose observation mode II that can be applied to the 3D human posture estimation model to characterize the local feature similarity of human posture and guide the design of the group connection module;

[0010] Step 4: Design and utilize the fully connected characteristics of the fully connected module to capture the global features of human posture, and obtain a high-dimensional vector representing the global features of human posture;

[0011] Step 5: Design and utilize the grouped local connection characteristics of the grouped connection module to capture the global features of the human body posture, and obtain a high-dimensional vector representing the local features of the human body posture;

[0012] Step 6: The fusion network fuses the parallel local features and global features and regresses them to 3D posture, finally obtaining a model for 3D human posture estimation.

[0013] As a further improvement of the present invention, it is characterized in that in step 1:

[0014] The input required by the algorithm is: the picture with human posture input to the algorithm, in the form of RGB;

[0015] The output required by the algorithm is: a two-dimensional vector representing the human body joint points, with a size of 17*2.

[0016] After the first stage of 2D pose estimation model training is completed, fine-tuning is performed on the existing model, and RGB images with human poses are used as input to obtain 2D key point information of human poses. This is combined with the 3D annotation information in the original data set to form 2D-3D paired key points as the second stage training data in the 3D human pose estimation model.

[0017] As a further improvement of the present invention, it is characterized in that in the step 2:

[0018] The distribution of the estimation error of the key points of the human body between each key point is not random, but shows a specific regularity. Observation mode I is the relationship between the estimation error of each key point of the human body and the human body structure. The error of the key point has a great correlation with the human body posture structure. The closer the key point is to the center of the human body (such as the hips, shoulders and spine), the smaller its estimation error; the farther the key point is from the center of the human body (such as the feet, wrists and head), the larger its error. Key points with small errors represent the global features of the human body posture and are related to all the key points of the human body posture; key points with high errors represent the local features of the human body posture and are only related to some key points. Based on mode I, the key points of the human body posture can be grouped for feature extraction and discovery of local posture.

[0019] As a further improvement of the present invention, it is characterized in that in step 3:

[0020] Mode II is the local features of similar human postures. Different actions of different human postures have high local feature similarity. Observation mode II is the relationship between human local features and posture estimation error. The local features of the human body have high similarity. Different local features always appear in different human postures. The higher the similarity of the local features of the posture, the smaller the estimation error; the lower the similarity of the local features of the posture, the larger the estimation error.

[0021] The pattern of local similar features of human postures shows that different cross-actions of human postures have high similarity in local features. Considering the structural and kinematic characteristics of human postures, local features are limited and appear repeatedly in different actions. For example, the local features of the lower body sitting posture appear in the action categories of making a phone call, smoking, eating, and sitting; the local features of the lower body standing posture also appear in actions such as discussing, indicating, and waiting. In addition, the local features are not strongly correlated with each other. For example, the postures of arms and legs are highly uncorrelated. Based on mode II, the local feature similarity pattern between the key points of human postures can group the key points for local feature mining. At the same time, different local features are highly uncorrelated, which provides a basis for parallel processing of different local features.

[0022] As a further improvement of the present invention, the specific method of step 4 is:

[0023] The algorithm of step 4 requires the following input: a set X of key points representing the 2D human body posture in 2D space.

[0024] The output required by the algorithm of step 4 is: a high-dimensional vector representing the global characteristics of human posture after overall encoding and connection through the fully connected network layer.

[0025] Step 4-1: Treat the set X of key points representing the 2D human body posture in the 2D space as a whole and directly stretch it into a representation vector.

[0026] Step 4-2: Pass the obtained representation vector through the fully connected encoder 1 to obtain a high-dimensional vector that encodes the global feature information of the human body posture.

[0027] Step 4-3: Connect the high-dimensional vector through multiple layers of the fully connected network layer, and finally obtain a feature vector representing the global information of the human body posture.

[0028] The global feature capture module is a fully connected network that learns the global features of human posture while processing the encoding vector representing the global information of the human body. It can be noted that in the global feature capture module, each output feature and each intermediate feature is arbitrarily connected to all input features, so as to learn the global information represented by each feature. Formally, the extraction of global features of human posture can be expressed as

[0029] F global =FCN(X)

[0030] Among them, X is a set of two-dimensional key points of the human body, with a total of N key points; FCN represents a global feature capture module composed of a fully connected network.

[0031] As a further improvement of the present invention, the specific method of step 5 is:

[0032] The algorithm of step 5 requires as input: a set X of key points representing 2D human body posture in 2D space.

[0033] The output required by the algorithm of step 5 is: a high-dimensional vector representing the local features of the human body posture after being connected by group local encoding and group connection network layers.

[0034] Step 5-1: Group the set X of key points representing 2D human body posture in 2D space into multiple individuals, and represent the groups as multiple representation vectors.

[0035] Step 5-2: Pass the obtained multiple representation vectors through the fully connected encoder 2 to obtain a high-dimensional vector that encodes the local feature information of the human body posture.

[0036] Step 5-3: The high-dimensional vector is connected through multiple layers of group connection network layers, and finally a feature vector representing the local information of the human body posture is obtained.

[0037] The local feature capture module is a group-connected network with a low-dimensional global context that learns the local features of human posture while processing the encoding vector representing the local information of the human group. Key points are grouped according to the similar patterns of the observed local features of the human body to capture the local features of the human posture. Formally, the extraction of local features of human posture can be expressed as

[0038]

[0039] Among them, X k It is a set of two-dimensional key points of the human body representing the group, with a total of k groups; GCN k represents the kth global feature capture module consisting of a fully connected network.

[0040] As a further improvement of the present invention, the specific method of step 6 is:

[0041] The input required by the algorithm of step 6 is: a feature vector representing the global information of the human body posture and a feature vector representing the local information of the human body posture.

[0042] The output required by the algorithm of step 6 is: a fusion vector of the feature vector of the global information of the human body posture and the feature vector of the local information of the human body posture, and the final three-dimensional representation of the human body posture.

[0043] After the representation of global features and local features is determined, a key issue is to fuse these two features. At this point, the global feature capture module and the local feature capture module can be placed in parallel or sequentially. Based on our work, parallel arrangement has better results than sequential arrangement, that is, learning the information of local features and global features separately, and then starting to fuse the two, and finally obtaining a model for 3D human posture estimation.

[0044] Beneficial effects:

[0045] 1. An observation model I is proposed in which the estimation error of each key point of the human body is highly correlated with the human body structure. The distribution of the estimation error of the key points of the human body between the key points is not random, but shows a specific regularity. Based on model I, the key points of the human body posture can be grouped for local posture feature extraction and discovery.

[0046] 2. A model II of local features with similar human postures is proposed. Different actions of different human postures have high local feature similarity. Based on model II, the local feature similarity patterns between key points of human postures can group key points for local feature mining. At the same time, different local features are highly uncorrelated, which provides a basis for parallel processing of different local features.

[0047] 3. Use the 2D pose detected by the off-the-shelf 2D pose estimator as the input of the proposed model method. The proposed method is compared with the advanced model methods using the standard subject protocol under protocol #1 and protocol #2. The experimental results show that the proposed method is 16.4mm (26.1% relative improvement) on average and performs better on most actions than the advanced model methods such as fully connected neural networks, graph neural networks and locally connected networks, indicating that the proposed model method has strong generalization ability for 3D human pose estimation.

[0048] 4. In the verification experiment, the cross-dataset generalization ability of the proposed model method is verified by training the model on the Human3.6M training set and evaluating it on the 3DPW test set. In the experimental results, this method achieves the best cross-dataset generalization performance.

[0049] In summary, the 3D human pose estimation method that integrates local and global features not only proposes observation mode I of the key point estimation error and the structural relationship of human pose and observation mode II of the local feature similarity in human pose, but also uses a parallel fusion network to extract and fuse local features and global features respectively, and obtains a 3D pose estimation model with excellent generalization performance. It has broad application prospects in human-computer interaction, virtual reality (VR), intelligent security and other fields. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] Figure 1 It is the overall flow chart of the present invention.

[0051] Figure 2 This is the observation mode I in step 2 of the present invention: the relationship between the estimation error of each key point of the human body and the human body structure. The error of the key point has a great correlation with the human body posture structure.

[0052] Figure 3 This is observation mode II in step 3 of the present invention: the local features of the human body have high similarity, and different local features always appear in different human body postures.

[0053] Figure 4 It is the global feature capture module in step 4 of the present invention, which is a fully connected network that learns the global features of human body posture when processing the encoding vector representing the global information of the human body.

[0054] Figure 5 It is the local feature capture module in step 5 of the present invention, which is a group connection network with a low-dimensional global context. When processing the encoding vector representing the local information of the human body group, it learns the local features of the human body posture.

[0055] Figure 6 It is the verification experiment result of the present invention, which shows that the present method not only generally outperforms advanced model methods such as fully connected neural networks, graph neural networks and locally connected networks, but also performs better in most actions. DETAILED DESCRIPTION

[0056] The following is a detailed description of a 3D human body posture estimation method that integrates local and global features proposed by the present invention in conjunction with the accompanying drawings:

[0057] As shown in the figure, a 3D human posture estimation method integrating local and global features includes:

[0058] Step 1: Adjust the image-2D posture network completed by training the cascade pyramid network, use the experimental data set to adjust the training on the trained model, and obtain the first stage model of the 3D human posture estimation model; obtain the 2D joint point coordinates of the human posture in the form of 16*2;

[0059] Step 2: Propose an observation mode I that can be applied to the 3D human posture estimation model to characterize the correlation between the error of the key points and the human posture, guide the design of the fully connected module, and stretch the human key points as a whole into a representation vector in the form of 32*1;

[0060] Step 3: Propose observation mode II that can be applied to the 3D human posture estimation model to characterize the local feature similarity of the human posture and guide the design of the grouping connection module. After grouping the human key points, they are represented as a group of representation vectors. The human key points can be divided into 2 groups, 3 groups or 5 groups.

[0061] The grouping of key points of human posture is determined based on the intuitive physical structure of the human body and data experiments. The final result is not only supported by experimental data, but also conforms to human intuitive cognition.

[0062] Step 4: Design and utilize the fully connected characteristics of the fully connected module to capture the global features of human posture, and obtain a high-dimensional vector representing the global features of human posture;

[0063] Step 4-1: The set X of key points representing the 2D human body posture in the 2D space is regarded as a whole and directly stretched into a representation vector. The transformation process from the set X to the representation vector is: 16*2 to 32*1.

[0064] Step 4-2: Pass the obtained representation vector through the fully connected encoder 1 to obtain a high-dimensional vector that encodes the global feature information of the human body posture. The dimension of the high-dimensional vector can be: 1024 dimensions, 2048 dimensions or 4096 dimensions.

[0065] Step 4-3: Connect the high-dimensional vector through multiple layers of the fully connected network layer, and finally obtain a feature vector representing the global information of the human body posture.

[0066] Step 5: Design and utilize the grouped local connection characteristics of the grouped connection module to capture the global features of the human body posture, and obtain a high-dimensional vector representing the local features of the human body posture;

[0067] Step 5-1: Group the set X of key points representing 2D human body posture in 2D space into multiple individuals. The groups are represented as multiple representation vectors. The sum of all key points in the multiple groups is still 16.

[0068] Step 5-2: Pass the obtained multiple representation vectors through the fully connected encoder 2 to obtain a high-dimensional vector that encodes the local feature information of the human body posture. Similarly, the dimension of the high-dimensional vector can be: 1024 dimensions, 2048 dimensions or 4096 dimensions.

[0069] Step 5-3: The high-dimensional vector is connected through multiple layers of group connection network layers to finally obtain a feature vector representing the local information of the human body posture. Similarly, the dimension of the high-dimensional vector representing the global feature information of the human body posture is the same as the sum of the dimensions of all high-dimensional vectors in multiple groups.

[0070] Step 6: The fusion network fuses the parallel local features and global features and regresses them to 3D posture, finally obtaining a model for 3D human posture estimation.

[0071] After the representation of global features and local features is determined, a key issue is to fuse these two features. At this point, the global feature capture module and the local feature capture module can be placed in parallel or sequentially. Based on our work, parallel arrangement has better results than sequential arrangement, that is, learning the information of local features and global features separately, and then starting to fuse the two, and finally obtaining a model for 3D human posture estimation.

[0072] The fusion network module fuses the feature vector of the global information of the human body posture and the feature vector of the local information of the human body posture, and then outputs a vector representing the characteristics of the human body posture. After that, it is transformed into the final 3D representation of the human body posture through the regression network module. The final 3D representation of the human body posture is 16*3.

[0073] The present invention not only proposes observation mode I of key point estimation error and structural relationship of human posture and observation mode II of local feature similarity in human posture, but also uses parallel fusion networks to extract and fuse local features and global features respectively, thus obtaining a 3D posture estimation model with excellent generalization performance, which has broad application prospects in human-computer interaction, virtual reality (VR), intelligent security and other fields.

[0074] The above content merely illustrates the technical idea of ​​the present invention and cannot be used to limit the protection scope of the present invention. Any changes made on the basis of this technical solution belong to the technical idea proposed by the present invention and fall within the protection scope of the claims of the present invention.

Claims

1. A 3D human body posture estimation method integrating local and global features, characterized in that: The specific steps are as follows: Step 1: Adjust the image-2D posture network completed by training the cascade pyramid network, use the experimental data set to adjust the training on the trained model, and obtain the first stage model of the 3D human posture estimation model; Step 2: Propose an observation mode I that can be applied to the 3D human posture estimation model to characterize the correlation between the error of key points and human posture, and guide the design of a fully connected module; Step 3: Propose observation mode II that can be applied to the 3D human posture estimation model to characterize the local feature similarity of human posture and guide the design of the group connection module; Step 4: Design and utilize the fully connected characteristics of the fully connected module to capture the global features of human posture, and obtain a high-dimensional vector representing the global features of human posture; Step 5: Design and utilize the grouped local connection characteristics of the grouped connection module to capture the global features of the human body posture, and obtain a high-dimensional vector representing the local features of the human body posture; Step 6: The fusion network fuses the parallel local features and global features and regresses them to 3D posture, finally obtaining a model for 3D human posture estimation; In step 4: The global feature capture module is a fully connected network that learns the global features of human posture when processing the encoding vector representing the global information of the human body. In the global feature capture module, each output feature and each intermediate feature are arbitrarily connected to all input features, so as to learn the global information represented by each feature. In form, the extraction of global features of human posture can be expressed as: F global =FCN(X) Among them, X is a set of two-dimensional key points of the human body, with a total of N key points; FCN represents a global feature capture module composed of a fully connected network; In step 5: The local feature capture module is a group-connected network with a low-dimensional global context. It learns the local features of human posture when processing the encoding vector representing the local information of the human group. According to the similar patterns of the observed local features of the human body, the key points are grouped to capture the local features of the human posture. In form, the extraction of local features of human posture can be expressed as: Among them, X k It is a set of two-dimensional key points of the human body representing the group, with a total of k groups; GCN k represents the kth global feature capture module consisting of a fully connected network.

2. A 3D human body posture estimation method integrating local and global features according to claim 1, characterized in that: In step 1: After the first stage of model training is completed, fine-tuning is performed, using RGB images with human posture as input to obtain 2D key point information of human posture, which is combined with the 3D annotation information in the original data set into 2D-3D paired key points and used as training data for the second stage in the 3D human posture estimation model.

3. A 3D human body posture estimation method integrating local and global features according to claim 1, characterized in that: In step 2: Observation mode I is the relationship between the estimation error of each key point of the human body and the human body structure. The error of the key point has a great correlation with the human body posture structure; The closer the key point is to the center of the human body, the smaller the estimation error is; the farther the key point is from the center of the human body, the larger the error is.

4. The method for 3D human body posture estimation by fusing local and global features according to claim 1, characterized in that: In step 3: Observation mode II is the relationship between the local features of the human body and the posture estimation error. The local features of the human body have high similarity, and different local features always appear in different human postures; the higher the similarity of the local features of the posture, the smaller the estimation error; the lower the similarity of the local features of the posture, the larger the estimation error.

5. The method for 3D human body posture estimation by fusing local and global features according to claim 1, characterized in that: In step 6: After the representation of global features and local features is determined, a key issue is to fuse these two features. At this time, the global feature capture module and the local feature capture module can be placed in parallel or sequentially. Parallel arrangement has better results than sequential arrangement. That is, the information of local features and global features is learned separately, and then the fusion of the two is started to finally obtain a model for 3D human posture estimation.

Citation Information

Cited By

  • 3D human body posture estimation method and system based on angle diagram and image feature fusion

    CN119888432A

  • A 3D human pose estimation method and system based on the fusion of angular graphs and image features

    CN119888432B