3D human pose estimation method based on complementary enhancement of key points and mesh vertices

Abstract

The present invention specifically relates to a 3D human posture estimation method based on complementary enhancement of key points and mesh vertices, including: outputting SMPL model parameters of each video frame through a regression prediction model; the processing steps of the regression prediction model include: extracting key point heat maps and shallow features and final layer features through a backbone network; converting the key point heat maps into 2D key point coordinates and 2D key point features in turn, extracting spatiotemporal features to generate 3D key point coordinates; generating partial 3D vertex heat maps through multi-scale feature processing and converting them into 3D vertex coordinates; generating key point-vertex coordinate embedding features, and outputting SMPL model parameters through an SMPL parameter regression module; constructing an SMPL model based on the SMPL model parameters of each video frame. The present invention can achieve continuous and smooth modeling estimation of a moving human body in continuous video frames, and can improve the problems of information loss and difficulty in fitting in the SMPL parameter regression network in the construction of limb rotation and body shape.

Description

Technical Field

[0001] The present invention relates to the technical field of computer vision, and in particular to a 3D human body posture estimation method based on complementary enhancement of key points and mesh vertices. Background Art

[0002] Human posture estimation is one of the most important tasks in computer vision in the past few decades and is the basis for machines to understand human movements. Its goal is to identify and locate all preset key points of the human body in images / videos (such as elbows, wrists, etc.) to digitize the posture and movement of the human body. This technology usually describes the posture with the coordinates of the key points of the human body or posture parameters, and can be applied to many technical fields such as human-computer interaction, virtual reality, intelligent monitoring, sports competition, etc.

[0003] In the past period of time, 2D human pose estimation has been the main research topic in the field of computer vision. However, 2D human pose estimation has some limitations: since the final estimation result is the coordinates of the key points of the human body on the two-dimensional image, it is impossible to accurately describe the specific position of the key points in the three-dimensional space and the rotation and shape information of the limbs. The goal of 3D human pose estimation is to infer the coordinates of the key points of the human body in the three-dimensional space to achieve a more accurate and realistic pose representation. Therefore, by converting the pose information into three-dimensional space, researchers can obtain more detailed information about the human body's pose, spatial structure, and motion state, providing broader possibilities for research and application in related fields.

[0004] The method of 3D reconstruction based on 2D detection results occupies an important position in 3D human pose estimation. The existing method first uses an effective 2D pose estimation network to output 2D human key point coordinates, and then inputs the 2D coordinates into a 3D reconstruction network to generate a reconstructed 3D pose. However, the 3D human key point coordinates also have their limitations. They only represent the position distribution of the human key points in space and cannot accurately describe the human body's posture in space. The existing method is to add an additional network of a regression parameterized human model so that the prediction result is converted from the 3D human key point coordinates to a human parameter model. Among them, the skinned multi-person linear model (SMPL) is a commonly used model in the human mesh reconstruction method. This model can obtain joint rotations, human body shape and joint coordinates. Under this model, the key point coordinates, body shape and limb rotation angles of the human body can be described in more detail.

[0005] When existing methods generate human body models through SMPL regression networks, the input information is usually the embedded features of 3D human body key point coordinates and the backbone network output feature information. However, existing image-based methods can reasonably predict the output from static images, but because they are designed and trained based on images, the inter-frame continuity in continuous video frames is destroyed, making it difficult to achieve continuous and smooth modeling estimation of moving human bodies in continuous video frames, resulting in poor continuity in 3D human body posture estimation. At the same time, the generated 3D human body key point coordinates can only accurately describe the position distribution of human body key points in space, but lack the description of limb rotation and body shape, which makes the SMPL parameter regression network have problems such as missing information and difficulty in fitting in the construction of limb rotation and body shape, resulting in the generated final model being inaccurate. Therefore, how to design a method that can improve the continuity and accuracy of 3D human body posture estimation is a technical problem that needs to be solved urgently. Summary of the invention

[0006] In view of the deficiencies of the above-mentioned prior art, the technical problem to be solved by the present invention is: how to provide a 3D human pose estimation method based on complementary enhancement of key points and mesh vertices, which can realize continuous and smooth modeling estimation of a moving human body in continuous video frames, and can improve the problems of information missing and fitting difficulty in the construction of limb rotation and body shape in the SMPL parameter regression network, thereby improving the continuity and accuracy of SMPL parameter regression and 3D human pose estimation.

[0007] In order to solve the above technical problems, the present invention adopts the following technical solutions:

[0008] The 3D human body posture estimation method based on complementary enhancement of key points and mesh vertices includes:

[0009] S1: Obtain continuous video frames to be estimated;

[0010] S2: Input the continuous video frames into the trained regression prediction model and output the SMPL model parameters of each video frame;

[0011] The processing steps of the regression prediction model are as follows:

[0012] S201: Input continuous video frames into the backbone network, extract key point heat maps of each video frame and shallow layer features and final layer features;

[0013] S202: sequentially converting the key point heat map of each video frame into 2D key point coordinates and 2D key point features; then learning the spatiotemporal correlation of the key points according to the 2D key point features of each video frame to generate corresponding 3D key point coordinates;

[0014] S203: performing multi-scale feature processing according to shallow layer features and final layer features of each video frame to generate a corresponding partial 3D vertex heat map; then converting the partial 3D vertex heat map of each video frame into corresponding 3D vertex coordinates;

[0015] S204: Generate key point-vertex embedding features according to the 3D key point coordinates and 3D vertex coordinates of each video frame;

[0016] S205: inputting the key point-vertex embedding features of each video frame into the SMPL parameter regression module, and outputting the corresponding SMPL model parameters;

[0017] S3: Construct a corresponding SMPL model based on the SMPL model parameters of each video frame as the 3D human posture model estimated for each video frame.

[0018] Preferably, data preprocessing is performed on the continuous video frames to obtain continuous video frames cropped for a single person, and then the continuous video frames cropped for a single person are input into the backbone network.

[0019] Preferably, the 3D key point coordinates of the video frame are generated by the following steps:

[0020] S2021: Convert the key point heat map of the video frame into the corresponding 2D key point coordinates through the βsoftargmax function;

[0021] S2022: Convert 2D keypoint coordinates into high-dimensional 2D keypoint features through linear embedding;

[0022] S2023: Input the high-dimensional 2D key point features into the spatiotemporal stacking attention module, and use the attention mechanism multiple times at the temporal level and spatial level to extract features between the same key points in different video frames and features between different key points in the same video frame to repeatedly learn the spatiotemporal correlation of the key points, and generate corresponding spatiotemporal weighted 3D key point features;

[0023] S2024: Generate corresponding 3D key point coordinates according to the spatiotemporal weighted 3D key point features.

[0024] Preferably, the spatiotemporal stacked attention module includes a plurality of stacked spatiotemporal attention modules; the features output by the previous spatiotemporal attention module are used as the input of the next spatiotemporal attention module, and the features output by the last spatiotemporal attention module are used as the spatiotemporal weighted 3D key point features;

[0025] Each spatiotemporal attention module includes the following processing steps:

[0026] 1) At the temporal level, the input features are first split at the key point level to decouple the key points, and then aggregated at the temporal level to obtain high-dimensional temporal features containing temporal information; secondly, the high-dimensional temporal features are layer-normalized to adjust the data distribution at the temporal level, and then further extract features through the MLP layer of the multi-layer perceptron structure; finally, after layer normalization, the multi-head attention module models the temporal correlation of the same key points, captures the motion trajectory of the key points, and generates features after temporal attention weighting;

[0027] 2) At the spatial level, the features weighted by temporal attention are decoupled at the temporal level and aggregated at the spatial level to generate high-dimensional spatiotemporal features containing spatial information; secondly, the high-dimensional spatiotemporal features are normalized at the spatial level to adjust the distribution of data, and then further extract features through the MLP layer of the multi-layer perceptron structure; finally, after layer normalization, the multi-head attention module models the spatial relationship between key points, captures the positional dependency between key points, and generates features weighted by spatiotemporal attention;

[0028] 3) Transform the feature dimension after spatiotemporal attention weighting to be consistent with the dimension of the input feature.

[0029] Preferably, the 3D vertex coordinates of the video frame are generated by the following steps:

[0030] S2031: Perform multi-scale feature extraction on shallow layer features and final layer features through the vertex regression module to obtain feature information of some mesh vertices, and then generate corresponding partial 3D vertex heat map H according to the feature information of some mesh vertices. 3D ;

[0031] S2032: multiplying a portion of the 3D vertex heat map by the weight coefficient and normalizing the map to generate a normalized 3D vertex heat map;

[0032] S2033: Convert the normalized 3D vertex heat map into corresponding 3D vertex coordinates through the βsoftargmax function.

[0033] Preferably, the vertex regression module includes a 3D convolutional layer, three stacked 3D residual blocks, a maximum pooling layer, a 3D residual block, a maximum pooling layer, a 3D residual block, a maximum pooling layer, two stacked 3D residual blocks, a 3D residual block, an upsampling layer, a 3D residual block, an upsampling layer, a 3D residual block, an upsampling layer, a 3D residual block, a 3D convolutional layer, and a 1×1×1 convolutional layer.

[0034] The 3D convolution layer consists of a 3D convolution layer, a batch normalization layer, and a ReLU activation function;

[0035] The 3D residual block first passes through a convolution layer, then 3D convolution and batch normalization, and finally the result is added to the input and input into the ReLU activation layer;

[0036] The max pooling layer performs a 3D max pooling operation with a stride of 2;

[0037] The upsampling layer consists of a 3D deconvolution layer with a stride of 2, a batch normalization layer, and a ReLU activation function.

[0038] Preferably, the 3D vertex coordinates are calculated and generated by the following formula:

[0039]

[0040]

[0041] Where: Represents the coordinates of the nth mesh vertex in three-dimensional space, that is, 3D vertex coordinates; 3D vertex heat map representing the nth mesh vertex; represents the 3D vertex heat map of the nth mesh vertex after normalization; ω represents the set weight coefficient; n represents the index number of the mesh vertex coordinate; r = [r x ,r y ,r z ], indicating the coordinates in three-dimensional space at (x, y, z).

[0042] Preferably, the 3D key point coordinates and 3D vertex coordinates of each video frame are first converted into corresponding 3D key point embedding features and 3D vertex embedding features through linear transformation; then the 3D key point embedding features and 3D vertex embedding features of each video frame are connected to form key point-vertex embedding features.

[0043] Preferably, the SMPL model parameters include the rotation parameter θ∈R 24×3 , body shape parameter β∈R 10 and camera parameters k c ∈R 3 .

[0044] Preferably, the regression prediction model is trained by the following loss function:

[0045] L=L pose +L mesh ;

[0046] L mesh =L param +L pose′ ;

[0047] Where: L represents the total loss of the regression prediction model; L poserepresents the L1 distance between the generated 3D keypoint coordinates and the label coordinates; L mesh represents the loss function for predicting SMPL parameters; L param represents the L1 distance between parameters θ and β and the label parameter; L pose′ Represents the L1 distance between the grid coordinates of the constructed SMPL model and the label.

[0048] Compared with the prior art, the 3D human body posture estimation method based on complementary enhancement of key points and mesh vertices in the present invention has the following beneficial effects:

[0049] The present invention predicts the SMPL model parameters of continuous video frames through a regression prediction model, and then constructs a 3D human posture model based on the SMPL model parameters. On the one hand, the spatiotemporal attention module of the regression prediction model of the present invention uses the attention mechanism distribution to extract the features between the same key points in different video frames and the features between different key points in the same video frame at the temporal level and the spatial level, respectively, so that the temporal consistency of the key point coordinates and the correlation between the key points (i.e., the inter-frame consistency of continuous video frames) can be effectively used to convert the 2D key points into 3D key point coordinates, and then continuous and smooth modeling estimation of the moving human body in continuous video frames can be achieved, thereby improving the continuity of SMPL parameter regression and 3D human posture estimation. On the other hand, based on the use of spatiotemporal attention for human posture estimation, the present invention extracts feature information of human mesh vertices through a vertex regression module and generates 3D vertex coordinates of each mesh vertex, so that the final SMPL model parameters can be regressed using 3D key point coordinates and 3D vertex coordinates, achieving complementary enhancement of key points and mesh vertices, supplementing the missing information such as limb rotation and human shape in the human key point skeleton, and can improve the problems of information missing and fitting difficulty in the construction of limb rotation and body shape in the SMPL parameter regression network, making the SMPL model regression more accurate and more in line with the image itself, and reducing the dependence on the feature extraction module, thereby improving the accuracy of SMPL parameter regression and 3D human posture estimation. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] In order to make the purpose, technical solution and advantages of the invention more clear, the present invention will be further described in detail below with reference to the accompanying drawings, in which:

[0051] Figure 1 It is a logic block diagram of the 3D human body posture estimation algorithm of the present invention;

[0052] Figure 2 This is the network structure diagram of the regression prediction model;

[0053] Figure 3 The network structure diagram of a single spatiotemporal attention module;

[0054] Figure 4 This is the network structure diagram of the vertex regression module;

[0055] Figure 5 It is the visualization result of the method of the present invention on 3DPW;

[0056] Figure 6 The visualization result of the method of the present invention on Huaman3.6;

[0057] Figure 7 Visual comparison results of VertAttPose (a) and MPSnet (b) on Huaman3.6;

[0058] Figure 8 Schematic diagram for mesh vertices visualization. DETAILED DESCRIPTION

[0059] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. The components of the embodiments of the present invention generally described and shown in the drawings here can be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of the present invention provided in the drawings is not intended to limit the scope of the claimed invention, but only represents selected embodiments of the present invention. Based on the embodiments in the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work belong to the scope of protection of the present invention.

[0060] It should be noted that similar numbers and letters represent similar items in the following drawings, so once an item is defined in one drawing, it does not need to be further defined and explained in the subsequent drawings. In the description of the present invention, it should be noted that the orientation or position relationship indicated by the terms "center", "upper", "lower", "left", "right", "vertical", "horizontal", "inside", "outside", etc. is based on the orientation or position relationship shown in the drawings, or the orientation or position relationship in which the invention product is usually placed when used, which is only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore cannot be understood as a limitation of the present invention. In addition, the terms "first", "second", "third", etc. are only used to distinguish the description, and cannot be understood as indicating or implying relative importance. In addition, the terms "horizontal", "vertical", etc. do not mean that the components are required to be absolutely horizontal or suspended, but can be slightly tilted. For example, "horizontal" only means that its direction is more horizontal than "vertical", and does not mean that the structure must be completely horizontal, but can be slightly tilted. In the description of the present invention, it is also necessary to explain that, unless otherwise clearly specified and limited, the terms "set", "install", "connect", and "connect" should be understood in a broad sense, for example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection, or it can be indirectly connected through an intermediate medium, or it can be the internal communication of two elements. For ordinary technicians in this field, the specific meanings of the above terms in the present invention can be understood according to specific circumstances.

[0061] The following is a further detailed description through specific implementation methods:

[0062] Example:

[0063] This embodiment discloses a 3D human body posture estimation method based on complementary enhancement of key points and mesh vertices.

[0064] like Figure 1 As shown, the 3D human body posture estimation method based on complementary enhancement of key points and mesh vertices includes:

[0065] S1: Obtain continuous video frames to be estimated;

[0066] S2: Input the continuous video frames into the trained regression prediction model and output the SMPL model parameters of each video frame;

[0067] Combination Figure 2 As shown, the processing steps of the regression prediction model are as follows:

[0068] S201: Input continuous video frames into the backbone network, extract key point heat maps of each video frame and shallow layer features and final layer features;

[0069] In this embodiment, the backbone network adopts the existing ResNet-50, whose network structure and training method are both existing technologies. It is also an existing mature method to extract the key point heat map of the video frame and the shallow features and final layer features through the backbone network. Among them, the backbone network is a deep neural network, whose shallow features mainly contain basic texture information, and the final layer features contain the key point feature information of the human body. The combination of shallow features and final layer features can effectively extract the feature information of the mesh vertex.

[0070] S202: sequentially converting the key point heat map of each video frame into 2D key point coordinates and 2D key point features; then learning the spatiotemporal correlation of the key points according to the 2D key point features of each video frame to generate corresponding 3D key point coordinates;

[0071] S203: performing multi-scale feature processing according to shallow layer features and final layer features of each video frame to generate a corresponding partial 3D vertex heat map; then converting the partial 3D vertex heat map of each video frame into corresponding 3D vertex coordinates;

[0072] S204: Generate key point-vertex embedding features according to the 3D key point coordinates and 3D vertex coordinates of each video frame;

[0073] In this embodiment, the 3D key point coordinates and 3D vertex coordinates of each video frame are first converted into corresponding 3D key point embedding features and 3D vertex embedding features through linear embedding; then the 3D key point embedding features and 3D vertex embedding features of each video frame are connected to form key point-vertex embedding features.

[0074] S205: inputting the key point-vertex embedding features of each video frame into the SMPL parameter regression module, and outputting the corresponding SMPL model parameters;

[0075] In this embodiment, the SMPL parameter regression module used is an existing SMPL parameter regressor (such as the SMPL parameter regressor disclosed in SUN K, XIAOB, LIU D, et al. Deep High-Resolution Representation Learning for Human Pose Estimation or LI C, LEE G H. From synthetic to real: Unsupervised Domain Adaptation for Animal Pose Estimation), which estimates the SMPL model parameters through the key point-vertex embedding feature F.

[0076] It should be noted that Figure 2 The keypoints shown in E are embedded in k and vertex embedding E v It can be understood as a feature after expanding information. The original coordinates only have 3-dimensional information: x, y, z. The embedded features expand the dimensions on this basis and mine the additional information through fitting training. This process is obtained through a simple fully connected layer.

[0077] S3: Construct a corresponding SMPL model based on the SMPL model parameters of each video frame as the 3D human posture model estimated for each video frame.

[0078] The present invention predicts the SMPL model parameters of continuous video frames through a regression prediction model, and then constructs a 3D human posture model based on the SMPL model parameters. On the one hand, the spatiotemporal attention module of the regression prediction model of the present invention uses the attention mechanism distribution to extract the features between the same key points in different video frames and the features between different key points in the same video frame at the temporal level and the spatial level, respectively, so that the temporal consistency of the key point coordinates and the correlation between the key points (i.e., the inter-frame consistency of continuous video frames) can be effectively used to convert the 2D key points into 3D key point coordinates, and then continuous and smooth modeling estimation of the moving human body in continuous video frames can be achieved, thereby improving the continuity of SMPL parameter regression and 3D human posture estimation. On the other hand, based on the use of spatiotemporal attention for human posture estimation, the present invention extracts feature information of human mesh vertices through a vertex regression module and generates 3D vertex coordinates of each mesh vertex, so that the final SMPL model parameters can be regressed using 3D key point coordinates and 3D vertex coordinates, achieving complementary enhancement of key points and mesh vertices, supplementing the missing information such as limb rotation and human shape in the human key point skeleton, and can improve the problems of information missing and fitting difficulty in the construction of limb rotation and body shape in the SMPL parameter regression network, making the SMPL model regression more accurate and more in line with the image itself, and reducing the dependence on the feature extraction module, thereby improving the accuracy of SMPL parameter regression and 3D human posture estimation.

[0079] The regression prediction model of the present invention was experimentally verified on 3DPW and Human3.6M. The results showed that the 3D human body model predicted by the present invention method is more accurate than that of the existing method. As a price, the model parameters and calculation amount are increased more than that of the existing method.

[0080] During the specific implementation process, the backbone network used is ResNet-50, and its framework is a single-person posture estimation network. Therefore, before the continuous video frames are input into the backbone network, it is necessary to perform data preprocessing on the continuous video frames, that is, to perform data preprocessing on the continuous video frames to obtain continuous video frames after single-person cropping, and then input the continuous video frames after single-person cropping into the backbone network.

[0081] Data preprocessing includes: for the same human target in consecutive video frames, positioning is performed through the human detection frame obtained from the current video frame (the corresponding human detection frame is generated by detecting the video frame through the human detector). In some detection results, the limbs (such as palms, feet, etc.) far away from the center of the human body cannot be completely included in the detection frame, resulting in incorrect estimation of the key points of this part. In order to avoid the above situation and ensure that the same human body in consecutive video frames has the same viewing angle, the human detection frame is enlarged by 25% so that the detection frame can completely contain the detected human body; then the enlarged human detection frame in the current frame is used to crop the consecutive multi-frame video frames to obtain the continuous video frames after single-person cropping, and then the continuous video frames after single-person cropping are sent to the backbone network to extract the key point heat map and key point features.

[0082] The present invention performs data preprocessing, i.e., video frame cropping, on continuous video frames to obtain continuous video frames after cropping of a single person, so that the human body located in the continuous video frames can be kept in the same area, which is conducive to the backbone network to better extract the key point heat map and shallow layer features and final layer features, thereby further improving the accuracy of subsequent SMPL parameter regression and 3D human posture estimation.

[0083] During the specific implementation process, the motivation for designing the spatiotemporal attention module is as follows: the existing methods mainly associate the 2D coordinates of all key points in space, because most of the adjacent key points exist on the same skeleton, and their positions will affect each other. However, after the input is converted from the original image to the key point coordinates, it is difficult for the network to establish a connection between individual coordinates, so the improvement of the single key point position information in space is limited. At the temporal level, the position of the same key point of the human body will not change suddenly, and it is often based on the position movement at the last moment. The time information can be used to correct the key point information, smooth the motion trajectory of the key point, and reduce the negative impression caused by the positioning errors of individual key points. Therefore, the spatiotemporal attention module of the present invention separately learns the embedded coordinate key point features at the time and space levels to capture the motion trajectories of different key points themselves.

[0084] The spatiotemporal attention module designed by the present invention is composed of a plurality of spatiotemporal attention modules stacked together, wherein a single spatiotemporal attention model is composed of a temporal attention module and a spatial attention module. The temporal attention module focuses on the attention between the same key points between different frames, and the spatial attention focuses on the attention between different key points in the same frame, and the two jointly focus on the temporal and spatial correlation of each key point. The temporal and spatial attention modules have the same attention structure, but due to the transformation of the input feature channel, the network can calculate different attention features.

[0085] The spatiotemporal stacking attention module generates the 3D keypoint coordinates of a video frame through the following steps:

[0086] S2021: Convert the key point heat map of the video frame into the corresponding 2D key point coordinates through the βsoftargmax function.

[0087] In this embodiment, the 2D key point coordinates of each video frame are represented by C N,K ∈R N×K×2 , N is the number of video frames, and K is the number of key points.

[0088] S2022: Convert 2D keypoint coordinates to high-dimensional 2D keypoint features through linear embedding (or fully connected layer) D m is the feature dimension; in this embodiment, linear embedding converts low-dimensional 2D coordinates into high-dimensional features, increases their information content, and facilitates subsequent processing.

[0089] S2023: Input the high-dimensional 2D key point features into the spatiotemporal stacking attention module, and use the attention mechanism multiple times at the temporal and spatial levels to extract features between the same key points in different video frames and features between different key points in the same video frames to repeatedly learn the spatiotemporal correlation of the key points and generate corresponding spatiotemporal weighted 3D key point features.

[0090] That is, the 2D key point features are weighted in time and space to generate 3D key point features. The attention mechanism used in this embodiment is an existing mature method.

[0091] The attention mechanism generally includes two steps: calculating the attention weight and applying the attention weight. When calculating the attention weight, various algorithms can be used, such as Softmax, Sigmoid, Max, etc. The calculated attention weight can be directly applied to the feature map, or re-weighted in the channel dimension. In this embodiment, the attention module used is also calculated based on the above two steps. The attention weight of the feature at each pixel position or channel is first calculated, and then the attention coefficient is assigned to the feature to obtain the result after attention weighting.

[0092] The input features first pass through three convolutional layers to obtain three features V, K, and Q, or a convolutional layer generates features that are three times the number of channels of the input tensor, and then splits them into three features V, K, and Q. V is used as the feature after further processing, and K and Q are used to calculate the similarity coefficient. The specific calculation process can be divided into two steps: first use K and Q to calculate the normalized attention coefficient, and then multiply V by the calculated attention coefficient to obtain the attention-weighted result. The process can be composed of the following formula:

[0093]

[0094]

[0095] S×V→F out ;

[0096] Where F in is the input feature, F out is the feature after attention weighting, T represents the transposition operation, where the shapes of V, K, and Q are the same as the input features. The above operation is a single attention calculation. Generally, single attention will focus on a certain feature. In practical applications, the feature points that the network should pay attention to are not limited to 1. For example, the detection target in human posture estimation is multiple key points of the human body. Multi-head attention is based on the above attention network structure, and uses the attention weight calculation module with the same structure but different parameters to calculate it multiple times. The purpose is to hope that the network can pay attention to different features in the input tensor. The head in the multi-head attention is the number of attention in the module, that is, the number of times the attention coefficient is calculated. The final output result weights and merges the attention results of multiple heads, so that the network is not limited to weighting the attention of a single feature in the end, but can extract more features and weight the attention of different features.

[0097] S2024: Generate corresponding 3D key point coordinates according to the spatiotemporal weighted 3D key point features.

[0098] In this embodiment, the 3D key point coordinates are first converted into a 3D key point heat map by a 1×1×1 convolution, and then the β softargmax function is used to generate the corresponding 3D key point coordinates from the 3D key point heat map.

[0099] Combination Figure 3 As shown, the spatiotemporal stacked attention module includes multiple stacked spatiotemporal attention modules; the features output by the previous spatiotemporal attention module are used as the input of the next spatiotemporal attention module, and the features output by the last spatiotemporal attention module are used as the spatiotemporal weighted 3D key point features;

[0100] Each spatiotemporal attention module includes the following processing steps:

[0101] 1) At the time level, firstly, the input feature X in Split at the key point level to decouple the key points, and then aggregate at the time level to obtain high-dimensional time features containing time information Secondly, the high-dimensional temporal features are normalized at the temporal level to adjust the distribution of the data, and then further extracted through the MLP layer of the multi-layer perceptron structure; finally, after layer normalization, the multi-head attention module models the temporal correlation of the same key points, captures the motion trajectory of the key points, and generates time-attention-weighted features.

[0102] 2) At the spatial level, the features weighted by temporal attention are decoupled at the temporal level and aggregated at the spatial level to generate high-dimensional temporal features containing spatial information. Secondly, the high-dimensional temporal features are normalized at the spatial level to adjust the distribution of data, and then further extract features through the MLP layer of the multi-layer perceptron structure; finally, after layer normalization, the multi-head attention module models the spatial relationship between key points, captures the positional dependency between key points, and generates the spatiotemporal attention weighted feature X out .

[0103] It should be noted that Figure 3 In the following formula, B represents batch, N represents the number of frames, K represents the number of key points, and D m Represents the dimensionality of the embedded features.

[0104] X in =B×N×K×D m ;

[0105] X t =(B×K)×N×D m ;

[0106] X out =B×N×K×D m ;

[0107] X s =(B×N)×K×D m .

[0108] 3) Feature X out The dimension is transformed to the same as the input feature X in The dimensions are consistent.

[0109] Existing methods only model a single time or space. Although spatiotemporal information is used, the network only focuses on using part of the feature information, and the use of information is not sufficient. The spatiotemporal attention module designed by the present invention is composed of multiple spatiotemporal attention modules stacked together. The temporal attention focuses on the attention between the same key points between different frames, and the spatial attention focuses on the attention between different key points in the same frame. The two jointly focus on the temporal and spatial correlation of each key point, so that the spatiotemporal correlation of the joint points can be repeatedly learned. At the same time, multiple spatiotemporal attention modules are used to extract features at the temporal and spatial levels at the same time, and the correlation between key points is fully utilized in time and space. Multiple modules are repeatedly stacked to extract features repeatedly, and the relevant features can be better obtained, thereby further improving the accuracy of subsequent SMPL parameter regression and 3D human posture estimation.

[0110] In the specific implementation process, the design motivation of the vertex regression module is as follows: the SMPL model essentially uses a grid composed of 3D vertices to describe the shape and posture of the human body, and the model itself is represented by model parameters. Most of the existing methods based on the SMPL model use 3D human key point coordinates to regress the SMPL model parameters. However, the generated 3D human key point coordinates can only accurately describe the position distribution of the human key points in space, but lack the description of limb rotation and body shape. This causes the SMPL parameter regression network to have problems such as missing information and difficulty in fitting in the construction of limb rotation and human shape. The usual practice is to increase the diversity and data of the data set or improve the SMPL model structure to enhance its fitting ability, but this will increase the cost of manual data processing and the amount of network parameters and training costs. In view of this, the present invention uses an additional multi-scale branch network to estimate some 3D vertex coordinates of the SMPL model. These vertices are used to supplement the missing body shape and limb rotation information in the human key points, so that the SMPL parameter regression module can be better learned.

[0111] The vertex regression module generates the 3D vertex coordinates of a video frame through the following steps:

[0112] S2031: Perform multi-scale feature extraction on shallow layer features and final layer features through the vertex regression module to obtain feature information of some mesh vertices (i.e., human body model vertices), and then generate corresponding partial 3D vertex heat map H according to the feature information of some mesh vertices 3D ;

[0113] In this embodiment, the existing mature means are used to extract multi-scale features from shallow features and final layer features. Among them, features of different scales, that is, different resolutions, low-resolution features are downsampled, and have strong semantic information representation capabilities, but the resolution of the feature map is low, and the representation capabilities of geometric information are weak (spatial geometric feature details are lacking); high-resolution features have strong geometric detail information representation capabilities, although the resolution is high, but the semantic information representation capabilities are weak. By combining the two, complementary enhancement can be achieved, and the ability of the network to extract information can be improved.

[0114] The number of 3D vertices is set according to actual needs.

[0115] S2032: Part of the 3D vertex heat map H 3D Multiply it by the weight coefficient ω and normalize it to generate a normalized 3D vertex heat map;

[0116] In this embodiment, the weight coefficient ω is set through experiments and can be set to 160.

[0117] S2033: Convert the normalized 3D vertex heat map into corresponding 3D vertex coordinates through the βsoftargmax function.

[0118] Combination Figure 4 As shown in Figure 3, the vertex regression module consists of a 3D convolutional layer, a 3D residual block, a maximum pooling layer, an upsampling layer, and a 1×1×1 convolutional layer, which includes 2 basic 3D convolutional layers, 11 3D residual blocks, 3 maximum pooling layers, 3 upsampling layers, and a 1×1×1 convolutional layer.

[0119] The vertex regression module includes a 3D convolutional layer connected end to end (end to end connection means that the output end of the previous module is connected to the input end of the next module), three stacked 3D residual blocks, a maximum pooling layer, a 3D residual block, a maximum pooling layer, a 3D residual block, a maximum pooling layer, two stacked 3D residual blocks, a 3D residual block, an upsampling layer, a 3D residual block, an upsampling layer, a 3D residual block, an upsampling layer, a 3D residual block, a 3D convolutional layer, and a 1×1×1 convolutional layer.

[0120] That is, the vertex regression module realizes multi-scale feature extraction and partial 3D vertex heat map generation through the above network structure.

[0121] The 3D convolution layer consists of a 3D convolution layer, a batch normalization layer, and a ReLU activation function;

[0122] The 3D residual block first passes through a convolution layer, then 3D convolution and batch normalization, and finally the result is added to the input and input into the ReLU activation layer;

[0123] The max pooling layer performs a 3D max pooling operation with a stride of 2;

[0124] The upsampling layer consists of a 3D deconvolution layer with a stride of 2, a batch normalization layer, and a ReLU activation function.

[0125] There are usually two ways to calculate the coordinates of the heat map: argmax operation and soft-argmax operation. The argmax function can accurately obtain the coordinates of the maximum value in the heat map, but it is a non-differentiable function and its gradient cannot be calculated. Therefore, end-to-end training cannot be achieved using this method. Soft-argmax normalizes the heat map, and then multiplies the normalized heat map with the corresponding coordinate value and sums them to get the coordinates of the original maximum value. Ideally, since the value at the original maximum value is close to 1 and the others are close to 0, only the coordinates at the maximum value are retained after multiplication and summation, and the others can be ignored because their values ​​are close to 0. Therefore, the coordinates of the maximum value can be calculated, and the function is differentiable. However, the maximum value in the actual predicted 3D heat map is not necessarily much larger than other values, so directly using soft-argmax is usually not accurate enough. Because the probability of the maximum value is not large enough, other larger values ​​will affect the calculation of the coordinates.

[0126] Since the input of the SMPL parameter regression module is usually the coordinates of the key points of the human body, the existing result is a three-dimensional vertex heat map of the mesh vertices. In order to enable the model to be trained end-to-end, the present invention uses a weighted 3D soft-argmax function to calculate the vertex coordinates, that is, the heat map is first multiplied by the weight coefficient ω to expand the gap between the maximum value and other values, and then the coordinates are obtained through the soft-argmax operation.

[0127] The 3D vertex coordinates are calculated using the following formula:

[0128]

[0129]

[0130] Where: Represents the coordinates of the nth mesh vertex in three-dimensional space, that is, 3D vertex coordinates; 3D vertex heat map representing the nth mesh vertex; represents the 3D vertex heat map of the nth mesh vertex after normalization; ω represents the set weight coefficient; n represents the index number of the mesh vertex coordinate; r = [r x ,r y ,r z ], indicating the coordinates in three-dimensional space at (x, y, z).

[0131] The vertex regression module of the present invention extracts feature information of human body mesh vertices according to key point features, and generates 3D vertex coordinates of each mesh vertex, so that the final SMPL model parameters can be regressed using the 3D key point coordinates and the 3D vertex coordinates, and the complementary enhancement of key points and mesh vertices is achieved, and the missing information such as limb rotation and human body shape in the human body key point skeleton is supplemented, which can improve the problems of information missing and fitting difficulty in the construction of limb rotation and body shape in the SMPL parameter regression network, so that the SMPL model regression is more accurate and more in line with the image itself, and the dependence on the feature extraction module is reduced.

[0132] In the specific implementation process, the SMPL model is no longer a human skeleton, but a skin model consisting of 6890 vertices. Its skin is a three-dimensional mesh, which is composed of 3 vertices connected in 3D space. Due to the large number of mesh vertices, direct regression is difficult, so the generated human body model is described using model parameters with less data.

[0133] The SMPL model parameters in the present invention include the rotation parameter θ∈R 24×3 , body shape parameter β∈R 10 and camera parameters k c ∈R 3 ; θ is used to describe the rotation angle of the child limb relative to the parent limb in the model; β is used to describe the shape and size of the human body, such as height, weight, etc.; k c Used for projection calculations.

[0134] The present invention uses model parameters with less data volume to describe the generated human body model, so that the regression target of the regression prediction model is converted from 6890 vertices to 85 actual parameters, which effectively reduces the computational complexity of the network.

[0135] In the specific implementation process, since the network first generates 3D key point coordinates and SMPL vertex coordinates, and then the two are used to generate SMPL model parameters through the SMPL regression module, the model parameters can generate the corresponding SMPL model after calculation. Therefore, the method of the present invention is supervised at two levels: 3D human key point coordinates and SMPL model parameters and mesh vertex coordinates.

[0136] The regression prediction model is trained using the following loss function:

[0137] L=L pose +L mesh ;

[0138] L mesh =L param +L pose′ ;

[0139] Where: L represents the total loss of the regression prediction model; L poserepresents the L1 distance between the generated 3D keypoint coordinates and the label coordinates; L mesh represents the loss function for predicting SMPL parameters; L param represents the L1 distance between parameters θ and β and the label parameter; L pose′ Indicates the L1 distance between the grid coordinates of the constructed SMPL model and the label. In the actual calculation process, 2D coordinates are used for calculation supervision, and the predicted key point coordinates are used to calculate the camera parameters k. c Projection onto a 2D plane.

[0140] The above-mentioned loss adopted by the present invention is supervised at two levels: key point coordinates and human body model. On the one hand, the key point coordinate level supervision ensures that the overall structure of the human body is clear, and at the same time ensures that the key point coordinates of the human body used to regress the human body model are accurate, avoiding the loss caused by wrong key point coordinates. On the other hand, the model parameters are supervised to ensure the accuracy of the final human body model generated, and then the human body model is projected onto a two-dimensional plane to supervise the 2D coordinates to ensure that the generated human body model fits the human body in the image, thereby helping to improve the accuracy of SMPL parameter regression and 3D human body posture estimation.

[0141] In order to better illustrate the advantages of the technical solution of the present invention, this embodiment discloses the following experiment.

[0142] 1. Experimental parameter settings

[0143] This experiment uses PyTorch on Ubuntu. The backbone network is the existing ResNet-50 (from HE K, ZHANG X, REN S, et al. Deep Residual Learning for Image Recognition), and the weights pre-trained on the ImageNet dataset are used as the initialization weights. The weights of the remaining parts are initialized with Gaussian distribution, o = 0.001. The weights are updated by Adam optimizer 22, and the mini batch size is 32. In order to crop the human body region from the input image, the ground truth bounding box is used in both training and testing stages. When there is no bounding box in the test stage, Mask R-CNN (from JAIN A, TOMPSON J, LECUN Y, et al. Modeep: A Deep Learning Framework Using Motion Features for Human Pose Estimation) is trained and tested to obtain the bounding box. The cropped human image is resized to 256x256.

[0144] Data augmentation was performed during training, including scaling (±25%), rotation (±60°), random horizontal flipping, and color jittering (±20%). The initial learning rate was 10-4. The regression prediction model of the present invention was trained for 6 epochs, and the learning rate was reduced by 10 times after the 3rd and 5th epochs. Similar to the existing method (from WEI WL, LIN JC, LIU TL, et al. Capturing Humans in Motion: Temporal-Attentive 3D Human Pose and Shape Estimation from Monocular Video), this experiment used MPJPE, PA-MPJPE, MPVPE, and ACC-ERR as evaluation indicators on the 3DPW dataset; since the official Human3.6M dataset does not give the SMPL model output (the existing data is mostly generated using network fitting or other means), the existing MPJPE, PA-MPJPE, and ACC-ERR are used as evaluation indicators.

[0145] 2. Comparison with existing methods

[0146] 3DPW dataset: This paper first shows the evaluation results using 3DPW as the test set, and the evaluation indicators are MPJPE, PA-MPJPE, MPVPE and ACC-ERR. Table 1 shows the results of quantitative comparison of the network designed in the present invention with the existing multi-frame based methods VIBE (from KOCABAS M, ATHANASIOU N, BLACK M J.Vibe:Video Inference for Human BodyPose and Shape Estimation), MEVA (from LUO Z, GOLESTANEH SA, KITANI K M.3dHumanMotion Estimation Via Motion Compression and Refinement), TCMR (from CHOI H, MOONG, CHANG JY, et al.Beyond Static Features for Temporally Consistent 3d HumanPose and Shape from a Video), and MPS-Net (from WEI WL, LIN JC, LIU TL, et al.Capturing Humans in Motion:Temporal-Attentive 3D Human Pose and ShapeEstimation from Monocular Video) on the 3DPW dataset. Figure 5 The visualization results of the method of the present invention under frontal view, sideways view, back view and occlusion on the 3DPW dataset are shown, and the comparison indicator is that it has higher accuracy in comparison.

[0147] Table 1 Quantitative comparison results of the method of the present invention and other methods on 3DPW

[0148]

[0149] Results based on the Human3.6M dataset: The model of the present invention was evaluated on the Human3.6M data, and the evaluation indicators were MPJPE, PA-MPJPE and ACC-ERR.

[0150] Table 2 shows the results of quantitative comparison between the proposed method and the existing multi-frame-based methods VIBE, MEVA, TCMR, and MPS-Net on Human3.6M. Figure 6 The visualization results of the method of the present invention under frontal view, lateral view, dorsal view and occlusion in the Human3.6M dataset are shown.

[0151] Table 2 Quantitative comparison results of the method of the present invention and different methods on Human3.6M

[0152]

[0153] The computational complexity and amount of the model of the present invention: Due to different algorithm platforms and most algorithms are not open source, the present invention cannot accurately know the operating efficiency of its model. Therefore, this experiment compares the model parameters and computational complexity to indirectly illustrate the cost. Table 3 shows the parameters and computational complexity of the method of the present invention and representative competitors VIBE, MEVA and TCMR. Since the network proposed by the present invention adds a vertex regression module compared to the previous network, the model parameters and computational complexity are more than the compared methods.

[0154] Table 3 Quantitative comparison results of the method of the present invention and different methods in floating point operations and parameter quantities

[0155]

[0156] In order to intuitively demonstrate the performance of the method of the present invention, Figure 7 The comparison with MPSnet (from WEI WL, LIN JC, LIU TL, et al. Capturing Humans in Motion: Temporal-Attentive 3D Human Pose and Shape Estimation from Monocular Video) at different viewing angles is shown. Compared with MPSnet, the result of the method of the present invention is more accurate: in the frontal perspective, MPSnet uses the coordinates of the key points of the human body as the feature regression model parameters. Since the key points lack rotation information, the estimated model footsteps present a strange posture. Similarly, in the front and back perspectives, the model posture estimated by the method proposed in the present invention is more consistent with the predicted human body, which shows that the additional vertex regression provides effective rotation information, and the complete information makes the prediction result more accurate.

[0157] 3. Ablation experiment

[0158] In order to reduce the training time, this experiment conducts an ablation experiment on the dataset 3DPW to verify the effectiveness of each module in the method of the present invention. The ablation module includes a vertex regression module and a temporal and spatial attention fusion. The experiment shows that the designed module is effective in improving the output accuracy.

[0159] Spatiotemporal attention module: In this ablation setting, this experiment explores the impact of different attention mechanisms on the final output posture. This experiment removes the temporal attention mechanism in the spatiotemporal module and only uses the spatial attention mechanism for feature extraction. From Table 4, we can see that ACC-ERR changes from 7.5 to 28.3. This is because the spatial attention only captures the direct dependency of key points in a single frame, ignoring the positional influence between key points at different times, which reduces the smoothness of the representation output. When the spatial attention mechanism is removed and only the temporal attention mechanism is used, all indicators decrease, and the network performance is better than the network with only single spatial attention. This is because the 2D coordinates extracted based on the 2D backbone network are used as input information. The 2D backbone network mainly extracts features at multiple scales for a single frame during extraction, and to a certain extent utilizes the spatial information between key points. Therefore, a single spatial attention mechanism has limited performance improvement on the network. Adding the temporal attention mechanism is essentially to use multi-frame information to smooth the output, reducing the impact of a certain 2D detection error.

[0160] At the same time, this experiment also explores the impact of the number of spatiotemporal attention modules stacked. The number of cascaded spatiotemporal attention modules is 2, 4, and 8. Each time the stacking is increased, the network performance will be improved to a certain extent. On the one hand, the increase in network parameters and the number of layers enhances the network's fitting ability. This also shows that repeated feature extraction can enable the data to establish the temporal and spatial dependencies of the intermediate features, thereby improving the final output accuracy.

[0161] Vertex Regression Module: In this setting, this experiment will explore the impact of the vertex regression module on network performance. Two settings are used in the experiment: removing the vertex regression module and choosing the number of vertices.

[0162] The results of removing the vertex regression module are shown in Table 4. The network using only 8 stacked spatiotemporal attention modules has a certain degree of decrease in accuracy compared to the complete network. This shows that the additional regressed vertices can supplement the regression of the subsequent SMPL parameters. At the same time, the impact of the number of vertex regression on network performance is also discussed. The number of regressed vertices is initially set to 34, and then gradually increased to twice the original until 392, and finally directly regressed all the vertices (6890) of the SMPL model. As can be seen from Table 4, when additional vertices are added, the network performance is improved, indicating that additional vertices can supplement the rotation and body shape information that the key points do not have. When the number of regressed vertices gradually increases, the network performance increases first and then decreases. This is because a small number of vertices can indeed supplement information, and the proposed vertex regression module can generate the corresponding vertices well. However, when the number of regressed vertices increases to all vertices, since the vertex regression module is a module with a small number of network parameters, its own structure and parameters cannot support its regression of such a large object, so the accuracy of its estimated vertices decreases, which in turn leads to a decrease in the final network prediction accuracy. This further illustrates that an appropriate number of vertices should be selected to supplement the information that the key points of the human body do not have, while the network itself can complete tasks of this complexity.

[0163] Table 4 Ablation studies performed on 3DPW by different modules in the method of the present invention

[0164]

[0165] Figure 8 The visualization results of the selected mesh vertices. It can be seen intuitively in the figure: 34, 68 vertices only cover a small part of the model, so their supplementary effect on missing information is limited; 136 vertices appear in all parts of the human body model, so their supplementary effect on information is better than the former; 272 vertices cover most of the human body, and the information is more complete than the former, but due to the large number, the designed vertex regression module is not enough to meet its needs, and additional calculations and parameters are required, so the prediction accuracy is reduced.

[0166] Finally, it should be noted that the above embodiments are only used to illustrate the technical solution of the present invention rather than to limit the technical solution. Those skilled in the art should understand that those modifications or equivalent substitutions of the technical solution of the present invention that do not depart from the purpose and scope of the technical solution should be included in the scope of the claims of the present invention.

Claims

1. A 3D human body posture estimation method based on complementary enhancement of key points and mesh vertices, characterized in that: include: S1: Obtain continuous video frames to be estimated; S2: Input the trained regression prediction model into the continuous video frames and output the skinned multi-person linear SMPL model parameters of each video frame; The processing steps of the regression prediction model are as follows: S201: Input continuous video frames into the backbone network, extract key point heat maps of each video frame and shallow layer features and final layer features; S202: sequentially converting the key point heat map of each video frame into 2D key point coordinates and 2D key point features; then learning the spatiotemporal correlation of the key points according to the 2D key point features of each video frame to generate corresponding 3D key point coordinates; In step S202, the 3D key point coordinates of the video frame are generated by the following steps: S2021: Convert the key point heat map of the video frame into the corresponding 2D key point coordinates through the βsoftargmax function; S2022: Convert 2D keypoint coordinates into high-dimensional 2D keypoint features through linear embedding; S2023: Input the high-dimensional 2D key point features into the spatiotemporal stacking attention module, and use the attention mechanism multiple times at the temporal level and spatial level to extract features between the same key points in different video frames and features between different key points in the same video frame to repeatedly learn the spatiotemporal correlation of the key points, and generate corresponding spatiotemporal weighted 3D key point features; S2024: Generate corresponding 3D key point coordinates according to the spatiotemporal weighted 3D key point features; S203: performing multi-scale feature processing according to shallow layer features and final layer features of each video frame to generate a corresponding partial 3D vertex heat map; then converting the partial 3D vertex heat map of each video frame into corresponding 3D vertex coordinates; S204: Generate key point-vertex embedding features according to the 3D key point coordinates and 3D vertex coordinates of each video frame; In step S204, firstly, the 3D key point coordinates and 3D vertex coordinates of each video frame are converted into corresponding 3D key point embedding features and 3D vertex embedding features through linear transformation; then, the 3D key point embedding features and 3D vertex embedding features of each video frame are connected to form key point-vertex embedding features; S205: inputting the key point-vertex embedding features of each video frame into the SMPL parameter regression module, and outputting the corresponding SMPL model parameters; In step S205, the SMPL model parameters include the rotation parameter θ∈R 24×3 , body shape parameter β∈R 10 and camera parameters k c ∈R 3 ; S3: Construct a corresponding SMPL model based on the SMPL model parameters of each video frame as the 3D human posture model estimated for each video frame.

2. The 3D human body posture estimation method based on complementary enhancement of key points and mesh vertices according to claim 1, characterized in that: In step S201, data preprocessing is performed on the continuous video frames to obtain continuous video frames after single-person cropping, and then the continuous video frames after single-person cropping are input into the backbone network.

3. The 3D human body posture estimation method based on complementary enhancement of key points and mesh vertices according to claim 1, characterized in that: In step S2023, the spatiotemporal stacking attention module includes a plurality of stacked spatiotemporal attention modules; the features output by the previous spatiotemporal attention module are used as the input of the next spatiotemporal attention module, and the features output by the last spatiotemporal attention module are used as the spatiotemporal weighted 3D key point features; Each spatiotemporal attention module includes the following processing steps: 1) At the temporal level, the input features are first split at the key point level to decouple the key points, and then aggregated at the temporal level to obtain high-dimensional temporal features containing temporal information; secondly, the high-dimensional temporal features are normalized at the temporal level to adjust the distribution of data, and then further extracted through the MLP layer of the multi-layer perceptron structure; Finally, after layer normalization, the multi-head attention module models the temporal correlation of the same key points, captures the motion trajectory of the key points, and generates features weighted by temporal attention. 2) At the spatial level, the features after time attention weighting are decoupled at the temporal level and aggregated at the spatial level to generate high-dimensional spatiotemporal features containing spatial information; secondly, the high-dimensional spatiotemporal features are normalized at the spatial level to adjust the distribution of data, and then further extract features through the MLP layer of the multi-layer perceptron structure; Finally, after layer normalization, the multi-head attention module models the spatial relationship between key points, captures the positional dependency between key points, and generates spatiotemporal attention weighted features; 3) Transform the feature dimension after spatiotemporal attention weighting to be consistent with the dimension of the input feature.

4. The 3D human body posture estimation method based on complementary enhancement of key points and mesh vertices according to claim 1, characterized in that: In step S203, the 3D vertex coordinates of the video frame are generated by the following steps: S2031: performing multi-scale feature extraction on shallow layer features and final layer features through a vertex regression module to obtain feature information of some mesh vertices, and then generating corresponding partial 3D vertex heat maps according to the feature information of some mesh vertices; S2032: multiplying a portion of the 3D vertex heat map by the weight coefficient and normalizing the map to generate a normalized 3D vertex heat map; S2033: Convert the normalized 3D vertex heat map into corresponding 3D vertex coordinates through the βsoftargmax function.

5. The 3D human body posture estimation method based on complementary enhancement of key points and mesh vertices according to claim 4, characterized in that: The vertex regression module includes a 3D convolutional layer connected end to end, three stacked 3D residual blocks, a maximum pooling layer, a 3D residual block, a maximum pooling layer, a 3D residual block, a maximum pooling layer, two stacked 3D residual blocks, a 3D residual block, an upsampling layer, a 3D residual block, an upsampling layer, a 3D residual block, an upsampling layer, a 3D residual block, a 3D convolutional layer, and a 1×1×1 convolutional layer. The 3D convolution layer consists of a 3D convolution layer, a batch normalization layer, and a ReLU activation function; The 3D residual block first passes through a convolution layer, then 3D convolution and batch normalization, and finally the result is added to the input and input into the ReLU activation layer; The max pooling layer performs a 3D max pooling operation with a stride of 2; The upsampling layer consists of a 3D deconvolution layer with a stride of 2, a batch normalization layer, and a ReLU activation function.

6. The 3D human body posture estimation method based on complementary enhancement of key points and mesh vertices according to claim 4, characterized in that: The 3D vertex coordinates are calculated using the following formula: Where: Represents the coordinates of the nth mesh vertex in three-dimensional space, that is, 3D vertex coordinates; 3D vertex heat map representing the nth mesh vertex; represents the 3D vertex heat map of the nth mesh vertex after normalization; ω represents the set weight coefficient; n represents the index number of the mesh vertex coordinate; r = [r x ,r y ,r z ], indicating the coordinates in three-dimensional space at (x, y, z).

7. The 3D human body posture estimation method based on complementary enhancement of key points and mesh vertices according to claim 1, characterized in that: The regression prediction model is trained using the following loss function: L=L pose +L mesh ; L mesh =L param +L pose′ ; Where: L represents the total loss of the regression prediction model; L pose represents the L1 distance between the generated 3D keypoint coordinates and the label coordinates; L mesh represents the loss function for predicting SMPL parameters; L param represents the L1 distance between parameters θ and β and the label parameter; L pose′ Represents the L1 distance between the grid coordinates of the constructed SMPL model and the label.

Images