Single-person 3D pose estimation method based on monocular video in occlusion scene

Abstract

The application provides a single-person 3D pose estimation method based on monocular video in an occlusion scene, and relates to the technical field of pose estimation. 2D key points are grouped by using human body prior knowledge, and data augmentation is performed by randomly occluding part of the 2D key point grouping in the model training stage to simulate human body pose data in an occlusion scene; a 3D human body pose estimation network architecture is designed by fusing a graph convolutional neural network (GCN) and a time convolutional neural network (TCN), the output of the GCN is used as the input of the TCN, and joint modeling of spatial and temporal information is realized. The method can not only effectively reduce the dependence on complex hardware, scene setting and manual labeling, but also flexibly generate various occlusion conditions, so that the data set can be quickly expanded, thereby greatly improving the data set construction efficiency; not only can the human body pose be accurately estimated in the occlusion scene, but also the shaking problem in the output video can be reduced to a certain extent, thereby improving the smoothness and stability of the estimation result.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of pose estimation, and particularly relates to a single-person 3D pose estimation method based on monocular video in a shielding scene. BACKGROUND

[0002] 3D human pose estimation refers to obtaining the joint points of a human body from given sensor input and positioning the positions of the joint points. 3D human pose estimation provides geometric and motion information of a human body and has been widely applied to fields such as motion analysis, virtual reality (VR), and healthcare.

[0003] 3D human pose estimation can be achieved by wearing sensors and other hardware or collecting human pose information from images and videos. Since the cost of sensors and other hardware devices is high, and image and video collection is relatively convenient, many works are currently committed to extracting human joint point coordinates and skeletal framework information from images or videos. In particular, with the rapid development of deep neural networks in recent years, human pose estimation based on vision has achieved high accuracy. For 3D human pose estimation of a single image, highly uncertain human motion, occlusion, and 2D to 3D ambiguity are all challenges faced. Current research has proven that there are two main aspects that affect the performance of the task, one is that there may be multiple 3D poses with the same 2D projection, and the other is that small errors in the estimation of 2D key points have a huge impact on 3D pose prediction. Therefore, 3D human pose estimation based on video sequences is more suitable, not only because video-like streaming data has gradually become an important form of data interaction in the current Internet and daily life, but also because the temporal information in the video brings new constraints to 3D human pose estimation. The method of introducing temporal information can effectively improve the performance of the algorithm.

[0004] Single-person 3D human pose estimation based on monocular video refers to using monocular video to infer the pose information of a single human body in three-dimensional space. Monocular video only provides information about the appearance and two-dimensional pose of the human body, and cannot directly obtain three-dimensional coordinates, so it is a challenging problem.

[0005] In practical applications, 3D human pose estimation faces many challenges, one of which is human occlusion. Human occlusion refers to the situation where part of the human body is occluded during video shooting, and the image containing the complete human body cannot be obtained. Human occlusion can be divided into three categories according to the source of occlusion: human self-occlusion, occlusion of other objects to the human body, and occlusion of other human bodies to the human body. In the case of no constraints on shooting scenes and characters, the above several types of human occlusion phenomena are inevitable. The input video cannot contain the complete human body, which brings great difficulty to the detection of 3D human joint points. First, the partial loss of the human body will make the algorithm unable to detect the occluded human joint points, so that the correct prediction of the occluded joint points cannot be made. Secondly, since the joint points of the human body are interrelated, the loss of part of the joint points will also affect the prediction of the unoccluded joint points. Therefore, not only does the algorithm require the correct detection of unoccluded joint points, but it also requires the ability to extract more favorable features to make reasonable predictions for the occluded joint points.

[0006] The production of 3D human pose datasets usually requires motion capture systems such as MoCap and wearable IMU. Due to this requirement, many 3D human pose datasets are produced in a restricted environment, and the scene of the dataset is relatively simple, lacking human pose annotations in occluded scenes. For example, the existing benchmark dataset Human3.6M contains almost no occluded scenes. Some current methods use game engines to produce synthetic datasets with occluded environments, which can solve the problem of insufficient occluded datasets to some extent. However, due to the domain gap problem between synthetic datasets and real datasets, deep models trained on synthetic datasets often cannot generalize well to real scenes, and it still requires a lot of effort to produce synthetic datasets.

[0007] Patent CN113361570B proposes a method of joint optimization of evolutionary data augmentation and network training based on reward and penalty function strategy. The distribution of data augmentation operations is synthesized through information in network training, and the evolutionary data augmentation algorithm is improved through the obtained enhancement distribution. Since this method relies on reward and penalty functions to dynamically adjust data augmentation operations, the calculation and training process can be complex, especially in the training of large-scale datasets or deep networks, which significantly increases the training time. Patent CN117911499A uses generated random masks to set some pixels of low-resolution images to 0 to simulate incomplete or noisy inputs. Random masks can generate invalid noise patterns, making some enhancement operations unable to effectively simulate noise or occlusion in real scenarios, affecting the effectiveness of model training. SUMMARY

[0008] The technical problem solved by the present application is to provide a single-person 3D pose estimation method based on monocular video in a shielding scene, aiming at the shortcomings of the prior art, which can not only accurately estimate the human body pose in a shielding scene, but also to some extent, reduce the jitter problem in the output video, thereby improving the smoothness and stability of the estimation result.

[0009] To solve the above technical problems, the technical solution adopted by the present application is:

[0010] A single-person 3D pose estimation method based on monocular video in a shielding scene, using human body prior knowledge to group 2D key points, and in the model training stage, through random shielding part of 2D key point grouping, data enhancement is carried out to simulate human body pose data in a shielding scene; a 3D human body pose estimation network architecture is designed, which integrates graph convolutional neural network GCN and time convolutional neural network TCN, wherein the GCN introduces a directed graph weight adjustment module, which is used for human body pose estimation in the spatial dimension; the TCN is based on mixed dilated convolution, which captures the dynamic changes between frames in the time dimension; the output of the GCN is used as the input of the TCN, realizing the joint modeling of spatial and temporal information.

[0011] Further, the simulation of human body pose data in a shielding scene specifically includes the following steps:

[0012] Step 1.1: 2D key point grouping;

[0013] In the Human3.6M dataset, the 17 joints of the human body are respectively 0-root node, 1-right hip, 2-right knee, 3-right ankle, 4-left hip, 5-left knee, 6-left ankle, 7-spine base, 8-chest, 9-neck, 10-head, 11-left shoulder, 12-left elbow, 13-left wrist, 14-right shoulder, 15-right elbow, 16-right wrist. First, based on human body prior knowledge, the 2D key points of the human body are grouped, and the grouping includes feet, knees, legs, left limbs and right limbs; the key point grouping G in the Human 3.6M dataset is as follows:

[0014] Feet: G1=[3,6],

[0015] Knees: G2=[2,3,5,6],

[0016] Legs: G3=[0,1,2,3,4,5,6],

[0017] Left side: G4=[1,2,3,14,15,16],

[0018] Right side: G5=[4,5,6,11,12,13];

[0019] Step 1.2: Set a random occlusion strategy; according to the occlusion characteristics of the application scene, set the occlusion probability P for different groups G in step 1.1, set P1=0.3, P2=0.3, P3=0.2, P4=0.1, P5=0.1; for each group of key points, generate a random number, if the random number is less than the occlusion probability of the group, occlude all 2D key points in the group, and fill the value with zero.

[0020] Further, in the construction of the directed graph, the directionality of the human skeleton is taken into account in the construction process of the graph, specifically as follows:

[0021] Step 2.1: Define the adjacency matrix, which is used to represent the connectivity between human key points; in the directed graph, the connection is one-way, and the adjacency matrix is asymmetric, A ij =1 represents a one-way connection from node i to node j; when constructing the directed graph, the connection direction in the adjacency matrix is adjusted according to the natural motion sequence of the human skeleton;

[0022] Step 2.2: Define the base feature matrix; the base feature matrix is the Euclidean distance between joints, which is used in combination with the adjacency matrix to define the connection between joints and provide additional weights or features for these connections; the base feature matrix is constructed with the Euclidean distance between the joints of the standard human body as the initial value; define the Euclidean distance d ij between joints, the calculation formula is as follows:

[0023]

[0024] A 17x17 base feature matrix is used to store the Euclidean distances between all joints.

[0025] Further, in step 2.1, the natural motion of the human skeleton is transmitted from top to bottom or from the torso to the limbs, so the matrix shown below is constructed to reflect this motion hierarchy:

[0026]

[0027] Further, the directed graph weight adjustment module implements the GCN anti-occlusion method based on directed graph weight adjustment, introduces node-level multi-head attention, and assigns different weights to different neighbor nodes to emphasize important neighbor information and let the model dynamically adjust the influence of neighbors on the target node; the specific method is as follows:

[0028] Step 3.1: Attention score calculation;

[0029] The attention score e ij is calculated by multiplying the node features with the learnable parameters aT Perform linear combination and apply activation function σ to calculate it. The calculation formula is as follows:

[0030]

[0031] Among them, e ij is the attention score of neighbor node j to node i, h i ||h j is the concatenation of the feature vectors of node i and node j; a T is a learnable weight vector used to parameterize the attention mechanism; σ is the ReLU activation function; d k is a normalization factor related to the degree of the node;

[0032] Step 3.2: Update the basic feature matrix F distance ;

[0033] For each pair of nodes i and j, F is updated based on the difference in the Euclidean distance between their feature vectors. distance ; F distance The update formula is as follows:

[0034] F distance (i,j)=exp(-||h i -h j ||)

[0035] Among them, ||h i -h j || is the Euclidean distance between the features of node i and node j, which is converted into a weight value close to the interval [0,1] through the exponential function;

[0036] Step 3.3: Node feature update;

[0037] Introducing a multi-head attention mechanism, each head uses a different weight matrix W i Transform the features of neighbor nodes and then use the attention score e ij Normalized weighted sum is used to aggregate these transformed features; while calculating the node feature update, the F updated in step 3.2 is distance Incorporated into the calculation process as edge weight; the new feature h of node i i It is constructed by connecting the results of all head outputs; the calculation formula for node feature update is as follows:

[0038]

[0039] Among them, softmax(e ij ) represents the attention score e of all neighbors j of each node i ij Normalize, Fdistance (i,j) provides dynamic weights based on feature differences; W i is a weight matrix used to transform the features h j of neighbor nodes j indicates that the layer uses K independent attention heads for feature aggregation, each head using a different weight matrix W k ; the results of each head aggregation are concatenated in the feature dimension, resulting in updated node features h i ;

[0040] Step 3.4: Node feature propagation;

[0041] Use the updated h i from step 3.3 as the new input features of each node, and the node feature propagation formula of each layer is as follows:

[0042] H (l+1) =σ(AH (l) W (l) )

[0043] where H (l) is the updated feature set of all nodes in the lth layer [h1, h2, …, h n ] in step 3.3, W (l) is the weight matrix of the lth layer, and σ is the ReLU activation function;

[0044] Step 3.5: Build GCN;

[0045] Each layer of the GCN updates the node features through the node feature propagation formula in step 3.4, thereby capturing the spatial dependencies between nodes; each GCN layer includes a graph convolution layer, a batch normalization layer, and a ReLU activation layer; the graph convolution layer is used to aggregate the features of adjacent nodes, the batch normalization layer is used to normalize the layer output, thereby speeding up the training process and improving the generalization ability of the model, and the ReLU activation layer introduces nonlinearity, enhancing the expression ability of the model.

[0046] Further, the time series convolutional neural network TCN is implemented by stacking multiple time series modules, residual connections are used between the multiple time series modules, and each time series module includes a hybrid dilated convolution, a batch normalization layer, a random deactivation layer, and a ReLU activation layer; the hybrid dilated convolution has an expansion coefficient set as a periodic sequence of [1, 2, 3] to increase the receptive field of the network.

[0047] Further, the specific process of fusing the graph convolutional neural network GCN and the time series convolutional neural network TCN is as follows:

[0048] Step 4.1: Model fusion; use the GCN constructed in step 3.5 to process the spatial relationships in the human body, extract node features at each time step; then, use the output of the GCN as the input of the TCN, and use the TCN to process the dynamic changes of the key points over time;

[0049] Step 4.2: Loss fusion; during model training, perform backpropagation and optimizer parameter update according to the total loss function; the specific method is as follows:

[0050] Step 4.2.1: Calculate the mean square error (MSE) to measure the error between the predicted key point positions and the true data, the calculation formula is as follows:

[0051]

[0052] Where, is the predicted key point coordinates of the model, J i is the corresponding true key point coordinates, and N is the total number of key points.

[0053] Step 4.3.2: Calculate the temporal consistency loss to punish the pose changes between adjacent frames in the time series, the calculation formula is as follows:

[0054]

[0055] Where, T is the total number of frames, and are the predicted key point coordinates at time t and t+1, respectively, t and J t+1 are the true key point coordinates at time t and t+1, respectively.

[0056] Step 4.3.3: Calculate the total loss; combine the two loss functions in steps 4.3.1 and 4.3.2 to form the final loss function, which considers both spatial accuracy and temporal continuity, the total loss calculation formula is as follows:

[0057] Total Loss = λ1·MSE Loss + λ2·Temporal Consistency Loss

[0058] Where, λ1 and λ2 are the weights used to balance the two loss components.

[0059] Further, in the model training, data in the public dataset Human3.6M in the field of human pose estimation is used, and neural network training is performed using the open source framework PyTorch; data enhancement is performed during training, the occlusion operation in step 1.2 is integrated into the training data generation pipeline, diversified training samples are generated through multiple random occlusions, the model encounters different occlusion conditions in each iteration, and thus learns how to rely on visible key points to accurately estimate 3D poses. SGD is used as the optimizer, the learning rate is set to 0.005, the dimension of the attention vector is set to 128, the number of attention heads is set to 10, and the epoch is set to 100.

[0060] The beneficial effects produced by the above technical solutions are that the single-person 3D pose estimation method based on monocular video in the occlusion scene provided by the application groups 2D key points based on human prior knowledge, and performs data enhancement in the training stage in the form of randomly shielding 2D key point groups to simulate human pose data in the occlusion scene. This method not only effectively reduces the dependence on complex hardware, scene setting and manual annotation, but also generates a variety of occlusion conditions flexibly, so that the dataset can be quickly expanded, thereby greatly improving the efficiency of dataset construction; the GCN anti-occlusion method based on directed graph weight adjustment is realized. Node-level multi-head attention is introduced. By assigning different weights to different neighbor nodes, important neighbor information is emphasized, and the model dynamically adjusts the influence of neighbors on the target node; combining the spatial modeling capability of GCN and the time series processing capability of TCN, the model can achieve better performance in the human pose estimation task. BRIEF DESCRIPTION OF DRAWINGS

[0061] Figure 1 The single-person 3D pose estimation method based on monocular video in the occlusion scene provided by the embodiment of the application is provided with a flowchart;

[0062] Figure 2 The 17 human key points in Human3.6M provided by the embodiment of the application are provided;

[0063] Figure 3 The network architecture diagram of the fusion GCN-TCN provided by the embodiment of the application is provided;

[0064] Figure 4 The effect comparison diagram after random occlusion enhancement provided by the embodiment of the application is provided; wherein, Fig. (4a) is an input image, Fig. (4b) is a prediction result without using the random occlusion method, Fig. (4c) is a prediction result using the random occlusion method, and Fig. (4d) is a true value. DETAILED DESCRIPTION

[0065] The specific embodiments of the present application are described in further detail below in conjunction with the accompanying drawings and examples. The following examples are used to illustrate the present application but are not intended to limit the scope of the present application.

[0066] As shown in FIG. 1, the method of the present embodiment is as follows. Figure 1

[0067] Step 1: Occlusion condition simulation. Group 2D keypoints based on human prior knowledge, and perform data augmentation in the training stage by randomly occluding 2D keypoint groups to simulate human pose data in occlusion scenarios.

[0068] Step 1.1: 2D keypoint grouping. In the Human3.6M dataset, the 17 joints of the human body are as shown in FIG. 2, which are 0-root node, 1-right hip, 2-right knee, 3-right ankle, 4-left hip, 5-left knee, 6-left ankle, 7-spine base, 8-chest, 9-neck, 10-head, 11-left shoulder, 12-left elbow, 13-left wrist, 14-right shoulder, 15-right elbow, and 16-right wrist. First, group the 2D keypoints of the human body based on human prior knowledge. Common groups include feet, knees, legs, left limbs, and right limbs, etc. For example, the keypoint group G in the Human 3.6M dataset is as follows: Figure 2 Feet: G1=[3, 6],

[0069] Knees: G2=[2, 3, 5, 6],

[0070] Legs: G3=[0, 1, 2, 3, 4, 5, 6],

[0071] Left side: G4=[1, 2, 3, 14, 15, 16],

[0072] Right side: G5=[4, 5, 6, 11, 12, 13].

[0073] Step 1.2: Random occlusion strategy setting. According to the occlusion characteristics of the application scenario, set the occlusion probability P for different groups G in step 1.1, set P1=0.3, P2=0.3, P3=0.2, P4=0.1, and P5=0.1. For each group of keypoints, generate a random number. If the random number is less than the occlusion probability of the group, mask all 2D keypoints in the group and fill them with zeros.

[0074]

[0075] ​​Step 1.3: Data augmentation during training. Integrate the masking operation from Step 1.2 into the training data generation pipeline by applying multiple random occlusion operations to generate diverse training samples. Use the augmented dataset for model training, where the model encounters different occlusion scenarios at each iteration, learning to infer how to rely on visible keypoints to accurately estimate 3D poses even with missing keypoint information.

[0076] Step 2: Directed graph construction. Incorporate the directionality of human body skeletons into the graph construction process. Traditional undirected graphs cannot capture the direction of motion between joints. In a directed graph, a directed graph is constructed by adjusting the adjacency matrix.

[0077] Step 2.1: Define the adjacency matrix. The adjacency matrix is used to represent the connectivity between human keypoints. In an undirected graph, the connection is symmetric, i.e., if there is a connection between node i and node j, then A ij = 1 and A ji = 1. However, in a directed graph, the connection is unidirectional, and the adjacency matrix is no longer symmetric, A ij = 1 represents a unidirectional connection from node i to node j. When constructing a directed graph, the direction of the connection in the adjacency matrix needs to be adjusted according to the natural motion sequence of the human skeleton. Generally, motion is transmitted from top to bottom or from the torso to the limbs, so the matrix is constructed as shown below to reflect this motion hierarchy.

[0078]

[0079] Step 2.2: Define the base feature matrix. The base feature matrix is the Euclidean distance between joints, which is used in conjunction with the adjacency matrix to define the connection between joints and provide additional weights or features for these connections. The base feature matrix is constructed using the Euclidean distance between each joint of the standard human body as the initial value. Define the Euclidean distance d ij between joints, which represents the distance between node i and node j, as follows:

[0080]

[0081] Store all joint Euclidean distances in a 17x17 base feature matrix F distance , where F distance is represented as follows:

[0082]

[0083] Step 3: Implement GCN Anti-occlusion method based on directed graph weight adjustment. Introduce node-level multi-head attention. By assigning different weights to different neighbor nodes, emphasize important neighbor information, let the model dynamically adjust the influence of neighbors on the target node.

[0084] Step 3.1: Attention score calculation. First, calculate the attention score e ij , the attention score is calculated by linear combination of node features and learnable parameters a T and applying activation function σ, the calculation formula is as follows:

[0085]

[0086] Where h i ||h j is the concatenation of the feature vectors of nodes i and j, a T is a learnable weight vector used to parameterize the attention mechanism, σ is the ReLU activation function, d k is the standardization factor related to the degree of the node.

[0087] Step 3.2: Update F distance . For each pair of nodes i and j, update F distance based on the Euclidean distance difference between the feature vectors. The F distance update formula is as follows:

[0088] F distance (i,j)=exp(-||h i -h j ||)

[0089] Where ||h i -h j || is the Euclidean distance between the features of nodes i and j, which is converted to a weight value close to the [0,1] interval through the exponential function.

[0090] Step 3.3: Node feature update. Introduce multi-head attention mechanism, each head uses a different weight matrix W i to transform the features of neighbor nodes, then use the weighted sum normalized by the attention score e ij to aggregate these transformed features. While calculating the node feature update, F distance in step 3.2 is integrated into the calculation process as edge weight. The new feature h i of node i is constructed by concatenating the results of all head outputs. The calculation formula of node feature update is as follows:

[0091]

[0092] where e ij is the attention score of neighbor node j to node i, softmax(e ij ) normalizes e ij for all neighbors j of each node i, F distance (i,j) provides dynamic weights based on feature difference, W i is the weight matrix used to transform the feature h j of neighbor node j, and σ is the ReLU activation function. denotes that the layer uses K independent attention heads for feature aggregation, each head using a different weight matrix W k . The results of each head aggregation are concatenated in the feature dimension, resulting in the updated node feature h i .

[0093] Step 3.4: Node feature propagation. Use h i obtained from step 3.3 as the new input feature of each node, and the node feature propagation formula of each layer is as follows:

[0094] H (l+1) = σ(AH (l) W (l) )

[0095] where H (l) is the updated feature set of all nodes in the l-th layer [h1, h2, …, h n ] in step 3.3, W (l) is the weight matrix of the l-th layer, and σ is the ReLU activation function.

[0096] Step 3.5: Building GCN. Each layer of GCN updates the node feature through the node feature propagation formula of the node in step 3.4, thereby capturing the spatial dependency between nodes. Each GCN layer includes a graph convolution layer, a batch normalization layer, and a ReLU activation layer. The graph convolution layer is used to aggregate the features of adjacent nodes, the batch normalization layer is used to normalize the layer output, thereby accelerating the training process and improving the generalization ability of the model, and the ReLU activation layer introduces nonlinearity to enhance the expression ability of the model.

[0097] Step 4: Fusion of GCN and TCN.

[0098] Step 4.1: Building TCN. TCN is implemented by stacking multiple temporal modules, and residual connections are used between multiple temporal modules. Each temporal module includes a mixed dilated convolution, a batch normalization layer, a random deactivation layer, and a ReLU activation layer; the dilated coefficients of the mixed dilated convolution are set to a periodic sequence of [1, 2, 3] to increase the receptive field of the network.

[0099] Step 4.2: Model fusion. The GCN constructed in Step 3 is used to process the spatial relationships in the human body, extracting node features at each time step. Then, the output of the GCN is taken as the input of the TCN, which is used to process the dynamic changes of the keypoints over time. The GCN-TCN network architecture diagram is shown in FIG. 4. Figure 3

[0100] Step 4.3: Loss fusion. During model training, the total loss function is used for backpropagation and parameter update of the optimizer.

[0101] Step 4.3.1: Calculate the mean square error (MSE). It is used to measure the error between the predicted keypoint positions of the model and the real data, and the calculation formula is as follows:

[0102]

[0103] where, is the predicted keypoint coordinates of the model, J i is the corresponding real keypoint coordinates, and N is the total number of keypoints, i.e., 17.

[0104] Step 4.3.2: Calculate the temporal consistency loss. To ensure that the predicted poses exhibit physical reasonableness and smoothness between consecutive frames, the temporal consistency loss is introduced to penalize the pose changes between adjacent frames in the time series, and the calculation formula is as follows:

[0105]

[0106] where T is the total number of frames, and are the predicted keypoint coordinates at time t and t+1, respectively, J t and J t+1 are the real keypoint coordinates at time t and t+1, respectively.

[0107] Step 4.3.3: Calculate the total loss. Combine the two loss functions in Step 4.3.1 and Step 4.3.2 to form the final loss function, which considers both spatial accuracy and temporal consistency, and the calculation formula is as follows:

[0108] Total Loss = λ1·MSE Loss + λ2·Temporal Consistency Loss

[0109] where λ1 and λ2 are the weights used to balance the two loss components.

[0110] Step 5: Model training.

[0111] ​In terms of data, the embodiment uses the data in the public dataset Human3.6M in the field of human pose estimation for training. In terms of training, the open source framework PyTorch is used for neural network training. Data augmentation is performed during training, and the occlusion operation in step 1.2 is integrated into the training data generation pipeline to generate diversified training samples through multiple random occlusions. The model encounters different occlusion conditions in each iteration, thereby learning how to rely on visible key points to accurately estimate 3D poses. The effect after random occlusion enhancement is compared as shown in FIGS. 4a-4d. Figure 4

[0112] The embodiment uses the public dataset Human3.6M in the field of human pose estimation for evaluation and comparison. As shown in Table 1, the method of the embodiment achieves higher accuracy compared with similar 3D human pose estimation methods. In Figure 2 the prediction effect under the condition of serious self-occlusion (SittingDown action) is further shown, and compared with other methods, the application can maintain higher accuracy.

[0113] Table 1 Performance comparison of the application and other methods on Human3.6M dataset

[0114]

[0115] In summary, the application proposes a single-person 3D pose estimation method based on monocular video in an occlusion scenario. Without manually constructing a human pose estimation dataset with occlusion conditions, various occlusion conditions can be flexibly generated in the training stage by randomly masking 2D key points, and the fused GCN and TCN model can still maintain high accuracy under the occlusion condition, which is conducive to the landing of human pose estimation technology in practical applications.

[0116] Finally, it should be noted that: the above embodiments are only used to illustrate the technical solutions of the application, and not to limit them; although the application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the foregoing embodiments, or make equivalent substitutions for part or all of the technical features; and these modifications or substitutions do not make the essence of the corresponding technical solutions deviate from the scope defined by the claims of the application.​

Claims

1. A method for single-person 3D pose estimation based on monocular video in occluded scenes, characterized by: 2D key points are grouped using prior knowledge of the human body. During the model training phase, data augmentation is performed by randomly occluding some of the 2D key points in the group to simulate human pose data in occluded scenarios. A 3D human pose estimation network architecture is designed that integrates a graph convolutional neural network (GCN) and a temporal convolutional neural network (TCN). The GCN introduces a directed graph weight adjustment module for human pose estimation in the spatial dimension. The TCN, based on mixed dilated convolution, captures dynamic changes between frames in the temporal dimension. The output of the GCN serves as the input of the TCN, achieving joint modeling of spatial and temporal information. The directed graph weight adjustment module implements a GCN anti-occlusion method based on directed graph weight adjustment. It introduces node-level multi-head attention and emphasizes important neighbor information by assigning different weights to different neighbor nodes, allowing the model to dynamically adjust the influence of neighbors on the target node. The specific method is as follows: Step 3.1: Attention score calculation; Attention score e ij By combining node features with learnable parameters a T Perform linear combination and apply activation function σ to calculate it. The calculation formula is as follows: Among them, e ij is the attention score of neighbor node j to node i, h i ||h j is the concatenation of the feature vectors of node i and node j; a T is a learnable weight vector used to parameterize the attention mechanism; σ is the ReLU activation function; d k is a normalization factor related to the degree of the node; Step 3.2: Update the basic feature matrix F distance ; For each pair of nodes i and j, F is updated based on the difference in Euclidean distance between their feature vectors. distance ; F distance The update formula is as follows: F distance (i,j)=exp(-||h i -h j ||) Among them, ||h i -h j || is the Euclidean distance between the features of node i and node j, which is converted into a weight value close to the interval [0,1] through the exponential function; Step 3.3: Node feature update; Introducing a multi-head attention mechanism, each head uses a different weight matrix W i Transform the features of neighbor nodes and then use the attention score e ij Normalized weighted sum is used to aggregate these transformed features; while calculating the node feature update, the F updated in step 3.2 is distance Incorporated into the calculation process as edge weights; the new feature h of node i i It is constructed by connecting the results of all head outputs; the calculation formula for node feature update is as follows: Among them, softmax(e ij ) represents the attention score e of all neighbors j of each node i ij Normalize, F distance (i, j) provides a dynamic weight based on feature differences; W i is the weight matrix used to transform the feature h of neighbor node j j ;σ is the ReLU activation function; Indicates that this layer uses K independent attention heads for feature aggregation, and each head uses a different weight matrix W k ; The results of each head aggregation will be spliced ​​in the feature dimension to obtain the updated node feature h i ; Step 3.4: Node feature propagation; Using the updated h obtained from step 3.3 i As the new input feature of each node, the node feature propagation formula of each layer is as follows: H (l+1) =σ(AH (l) W (l) ) Among them, H (l) is the feature set of all nodes in the lth layer after update in step 3.3 [h1,h2,…,h n ],W (l) is the weight matrix of the lth layer, σ is the ReLU activation function; Step 3.5: Build GCN; Each layer of GCN updates node features through the node feature propagation formula in step 3.4 to capture the spatial dependencies between nodes. Each GCN layer includes a graph convolution layer, a batch normalization layer, and a ReLU activation layer. The graph convolution layer is used to aggregate the features of adjacent nodes, the batch normalization layer is used to normalize the layer output, thereby accelerating the training process and improving the generalization ability of the model. The ReLU activation layer introduces nonlinearity to enhance the expressiveness of the model.

2. The method for single-person 3D pose estimation based on monocular video in occluded scenes according to claim 1, characterized in that: The simulation of human body posture data in the occlusion scene specifically includes the following steps: Step 1.1: 2D keypoint grouping; In the Human3.6M dataset, the 17 joints of the human body are 0-root node, 1-right hip, 2-right knee, 3-right ankle, 4-left hip, 5-left knee, 6-left ankle, 7-base of spine, 8-chest, 9-neck, 10-head, 11-left shoulder, 12-left elbow, 13-left wrist, 14-right shoulder, 15-right elbow, and 16-right wrist. First, based on prior knowledge of the human body, the 2D key points of the human body are grouped into two groups: feet, knees, legs, left limbs, and right limbs. The key point grouping G in the Human 3.6M dataset is as follows: Both feet: G1 = [3, 6], Both knees: G2 = [2, 3, 5, 6], Legs: G3 = [0, 1, 2, 3, 4, 5, 6], Left: G4 = [1, 2, 3, 14, 15, 16], Right side: G5 = [4, 5, 6, 11, 12, 13]; Step 1.2: Set a random occlusion strategy; according to the occlusion characteristics of the application scenario, set the occlusion probability P for different groups G in step 1.

1. The occlusion probabilities corresponding to the five groups are set to P1=0.3, P2=0.3, P3=0.2, P4=0.1, and P5=0.1 respectively; for each group of key points, generate a random number. If the random number is less than the occlusion probability of the group, occlude all 2D key points of the group and fill its values ​​with zero.

3. The method for single-person 3D pose estimation based on monocular video in occluded scenes according to claim 2, characterized in that: In the construction of the directed graph, the directionality of the human skeleton is incorporated into the graph construction process, as follows: Step 2.1: Define the adjacency matrix to represent the connectivity between key points of the human body; in a directed graph, the connection is unidirectional, the adjacency matrix is ​​asymmetric, A ij =1 indicates a unidirectional connection from node i to node j; when constructing a directed graph, the connection direction in the adjacency matrix is ​​adjusted according to the natural movement order of the human skeleton; Step 2.2: Define the basic feature matrix; The basic feature matrix is ​​the Euclidean distance between joints. It is used in conjunction with the adjacency matrix to define the connections between joint points. At the same time, the basic feature matrix provides additional weights or features for these connections. The Euclidean distance between the joints of the standard human body is used as the initial value to construct the basic feature matrix; Define the Euclidean distance d between joints ij , represents the distance between node i and node j, and the calculation formula is as follows: Using a 17×17 basic feature matrix To store the Euclidean distances between all joints.

4. The method for single-person 3D pose estimation based on monocular video in occluded scenes according to claim 3, characterized in that: In step 2.1, the natural movement of the human skeleton is transmitted from top to bottom or from the trunk to the limbs, so the following matrix is ​​constructed to reflect this movement hierarchy:

5. The method for single-person 3D pose estimation based on monocular video in occluded scenes according to claim 4, characterized in that: The temporal convolutional neural network (TCN) is implemented by stacking multiple temporal modules. Residual connections are used between the multiple temporal modules. Each temporal module includes a mixed dilated convolution, a batch normalization layer, a random dropout layer, and a ReLU activation layer. The dilation coefficient of the mixed dilated convolution is set to a periodic sequence of [1, 2, 3] to increase the receptive field of the network.

6. The method for single-person 3D pose estimation based on monocular video in occluded scenes according to claim 5, characterized in that: The specific process of fusing the graph convolutional neural network GCN and the temporal convolutional neural network TCN is as follows: Step 4.1: Model fusion; use the GCN constructed in step 3.5 to process the spatial relationships in the human body and extract the node features at each time step; then, use the output of the GCN as the input of the TCN, and use the TCN to process the dynamic changes of key points over time; Step 4.2: Loss fusion: During model training, perform backpropagation and optimizer parameter updates based on the total loss function. The specific method is as follows: Step 4.2.1: Calculate the mean square error (MSE), which is used to measure the error between the key point positions predicted by the model and the actual data. The calculation formula is as follows: in, is the key point coordinate predicted by the model, J i is the corresponding real key point coordinate, N is the total number of key points; Step 4.3.2: Calculate the temporal continuity loss to penalize the pose changes between adjacent frames in the time series. The calculation formula is as follows: Where T is the total number of frames, and The predicted key point coordinates at time t and t+1, J t and J t+1 are the real key point coordinates at time t and t+1 respectively; Step 4.3.3: Calculate the total loss; combine the two loss functions of step 4.3.1 and step 4.3.2 to form the final loss function, taking into account both spatial accuracy and temporal continuity. The total loss calculation formula is as follows: Total Loss =λ1·MSE Loss +λ2·Temporal Consistency Loss Among them, λ1 and λ2 are weights used to balance the two loss components.

7. The method for single-person 3D pose estimation based on monocular video in occluded scenes according to claim 6, characterized in that: The model training used data from the public dataset Human3.6M for human pose estimation, and the open-source framework PyTorch was used for neural network training. Data augmentation was performed during training, integrating the occlusion operation in step 1.2 into the training data generation pipeline. Diverse training samples were generated through multiple random occlusions. The model encountered different occlusion situations in each iteration, thereby learning how to accurately estimate 3D poses based on visible key points. SGD is used as the optimizer, the learning rate is set to 0.005, the dimension of the attention vector is set to 128, the number of attention heads is set to 10, and the epoch is set to 100.

Images