Single-person 3D attitude estimation method based on monocular video and oriented to occlusion scene

Abstract

The invention provides a single-person 3D attitude estimation method based on a monocular video for an occlusion scene, and relates to the technical field of attitude estimation. Human body priori knowledge is utilized to group the 2D key points, data enhancement is performed in a mode of randomly shielding part 2D key point grouping in a model training stage, and human body posture data in a shielding scene is simulated; a 3D human body posture estimation network architecture fusing a graph convolutional neural network GCN and a time sequence convolutional neural network TCN is designed, the output of the GCN serves as the input of the TCN, and joint modeling of space and time information is achieved. According to the method, the dependence on complex hardware, scene setting and manual annotation is effectively reduced, and the data set can be quickly expanded by flexibly generating various shielding conditions, so that the data set construction efficiency is greatly improved; the human body posture can be accurately estimated in a shielding scene, and the jitter problem in an output video is relieved to a certain extent, so that the smoothness and the stability of an estimation result are improved.

Description

Technical Field

[0001] The present invention relates to the technical field of posture estimation, and in particular to a single-person 3D posture estimation method based on monocular video in an occluded scene. Background Art

[0002] 3D human pose estimation refers to obtaining the joints of the human body from given sensor inputs and locating the positions of each joint. 3D human pose estimation provides geometric and motion information of the human body and has been widely used in motion analysis, virtual reality (VR), healthcare and other fields.

[0003] 3D human pose estimation can be achieved by collecting human pose information through hardware such as wearable sensors or from images and videos. Since hardware equipment such as sensors is expensive, while image and video acquisition is relatively convenient, there are many works devoted to extracting human joint coordinates and skeleton frame information from images or videos. In particular, with the rapid development of deep neural networks in recent years, human pose estimation based on vision has achieved high accuracy. For 3D human pose estimation of a single image, highly uncertain human motion, occlusion, and ambiguity from 2D to 3D are all challenges it faces. Current research has proved that there are two main aspects that affect the performance of this task. One is that there may be multiple 3D poses with the same 2D projection, and the other is that the misestimation of tiny 2D key points has a huge impact on 3D pose prediction. Therefore, 3D human pose estimation based on video sequences is more suitable, not only because video stream data has gradually become an important form of data interaction between the Internet and daily life, but also because the temporal information in the video brings new constraints to 3D human pose estimation. The method of introducing temporal information can effectively improve the performance of the algorithm.

[0004] Single-person 3D human pose estimation based on monocular video refers to using monocular video to infer the pose information of a single person in three-dimensional space. Monocular video only provides information about the appearance and two-dimensional pose of the human body, but cannot directly obtain three-dimensional coordinates, so this is a challenging problem.

[0005] In practical applications, 3D human pose estimation faces many challenges, one of which is human occlusion. Human occlusion means that part of the human body is blocked during video shooting, and an image containing the complete human body cannot be obtained. Human occlusion can be divided into three categories according to the source of occlusion: human body occlusion, other objects occluding the human body, and other human bodies occluding the human body. In the absence of any constraints on the shooting scene and characters, the above different types of human occlusion phenomena are inevitable. The input video cannot contain a complete human body, which brings great difficulties to the detection of 3D human joints. First, the partial loss of the human body will make the algorithm unable to detect the occluded human joints, and thus cannot give a correct prediction of the occluded joints. Secondly, since the joints of the human body are interrelated, the loss of some joints will also affect the prediction of the unoccluded joints. Therefore, the algorithm is not only required to correctly detect the unoccluded joints, but also to be able to extract more favorable features and make reasonable predictions for the occluded joints.

[0006] The production of 3D human pose datasets usually requires motion capture systems such as MoCap and wearable IMU. Due to this requirement, many 3D human pose datasets are produced in restricted environments, with relatively simple dataset scenes and lack of human pose annotations in occluded scenes. For example, the existing benchmark dataset Human3.6M contains almost no occluded scenes. At present, some methods use game engines to produce synthetic datasets with occluded environments. Although this can solve the problem of insufficient occluded datasets to a certain extent, due to the domain gap between synthetic datasets and real datasets, deep models trained on synthetic datasets often cannot generalize well to real scenes, and it still takes a lot of effort to produce synthetic datasets.

[0007] Patent CN113361570B proposes a method for jointly optimizing evolutionary data enhancement and network training based on a reward and penalty function strategy. The distribution of data enhancement operations is synthesized through information in network training, and the evolutionary data enhancement algorithm is improved through the obtained enhancement distribution. Since this method relies on reward and penalty functions to dynamically adjust data enhancement operations, the calculation and training process may be more complicated, especially in the training of large-scale data sets or deep networks, resulting in a significant increase in training time. Patent CN117911499A uses a generated random mask to set part of the pixels of a low-resolution image to 0, simulating incomplete or noisy inputs. Random masks may generate invalid noise patterns, resulting in some enhancement operations that cannot effectively simulate noise or occlusion in real scenes, affecting the effectiveness of model training. Summary of the invention

[0008] The technical problem to be solved by the present invention is to address the deficiencies of the above-mentioned prior art and provide a single-person 3D posture estimation method based on monocular video in occluded scenes, which can not only accurately estimate the human body posture in occluded scenes, but also alleviate the jitter problem in the output video to a certain extent, thereby improving the smoothness and stability of the estimation results.

[0009] In order to solve the above technical problems, the technical solution adopted by the present invention is:

[0010] A single-person 3D pose estimation method based on monocular video in occluded scenes uses prior knowledge of the human body to group 2D key points, and performs data enhancement by randomly occluding some 2D key points in the model training stage to simulate the human pose data in occluded scenes. A 3D human pose estimation network architecture that integrates a graph convolutional neural network (GCN) and a temporal convolutional neural network (TCN) is designed, in which a directed graph weight adjustment module is introduced into GCN to estimate human pose in the spatial dimension. TCN captures dynamic changes between frames in the temporal dimension based on mixed dilated convolution. The output of GCN is used as the input of TCN to achieve joint modeling of spatial and temporal information.

[0011] Furthermore, the simulation of human body posture data in an occlusion scene specifically includes the following steps:

[0012] Step 1.1: 2D key point grouping;

[0013] In the Human3.6M dataset, the 17 joints of the human body are 0-root node, 1-right hip, 2-right knee, 3-right ankle, 4-left hip, 5-left knee, 6-left ankle, 7-base of spine, 8-chest, 9-neck, 10-head, 11-left shoulder, 12-left elbow, 13-left wrist, 14-right shoulder, 15-right elbow, 16-right wrist. First, based on the prior knowledge of the human body, the 2D key points of the human body are grouped, including feet, knees, legs, left limbs and right limbs; the key point grouping G in the Human 3.6M dataset is as follows:

[0014] Both feet: G1 = [3,6],

[0015] Double knees: G2 = [2,3,5,6],

[0016] Legs: G3 = [0, 1, 2, 3, 4, 5, 6],

[0017] Left side: G4 = [1,2,3,14,15,16],

[0018] Right side: G5 = [4,5,6,11,12,13];

[0019] Step 1.2: Set a random occlusion strategy; according to the occlusion characteristics of the application scenario, set the occlusion probability P for different groups G in step 1.1, set P1 = 0.3, P2 = 0.3, P3 = 0.2, P4 = 0.1, P5 = 0.1; for each group of key points, generate a random number. If the random number is less than the occlusion probability of the group, occlude all 2D key points of the group and fill their values ​​with zero.

[0020] Furthermore, in the construction of the directed graph, the directionality of the human skeleton is incorporated into the graph construction process, as follows:

[0021] Step 2.1: Define the adjacency matrix to represent the connectivity between key points of the human body; in a directed graph, the connection is unidirectional, the adjacency matrix is ​​asymmetric, A ij =1 indicates a unidirectional connection from node i to node j; when constructing a directed graph, the connection direction in the adjacency matrix is ​​adjusted according to the natural movement order of the human skeleton;

[0022] Step 2.2: Define the basic feature matrix; the basic feature matrix is ​​the Euclidean distance between joints, which is used in conjunction with the adjacency matrix to define the connections between joints, and at the same time provide additional weights or features for these connections through the basic feature matrix; use the Euclidean distance between the joints of the standard human body as the initial value to construct the basic feature matrix; define the Euclidean distance between joints d ij , represents the distance between node i and node j, and the calculation formula is as follows:

[0023]

[0024] Using a 17×17 basic feature matrix To store the Euclidean distances between all joints.

[0025] Furthermore, in step 2.1, the natural movement of the human skeleton is transmitted from top to bottom or from the trunk to the limbs, so the following matrix is ​​constructed to reflect this movement hierarchy:

[0026]

[0027] Furthermore, the directed graph weight adjustment module implements a GCN anti-occlusion method based on directed graph weight adjustment, introduces node-level multi-head attention, and emphasizes important neighbor information by assigning different weights to different neighbor nodes, so that the model can dynamically adjust the influence of neighbors on the target node; the specific method is as follows:

[0028] Step 3.1: Attention score calculation;

[0029] Attention score ij By combining the node features with the learnable parameters aT Perform a linear combination and apply the activation function σ to calculate it. The calculation formula is as follows:

[0030]

[0031] Among them, e ij is the attention score of neighbor node j to node i, h i ||h j is the concatenation of the feature vectors of node i and node j; a T is a learnable weight vector used to parameterize the attention mechanism; σ is the ReLU activation function; d k is a normalization factor related to the degree of the node;

[0032] Step 3.2: Update the basic feature matrix F distance ;

[0033] For each pair of nodes i and j, F is updated based on the difference in the Euclidean distance between their feature vectors. distance ; F distance The update formula is as follows:

[0034] F distance (i,j)=exp(-||h i -h j ||)

[0035] Among them, ||h i -h j || is the Euclidean distance between the features of node i and node j, which is converted into a weight value close to the interval [0,1] through the exponential function;

[0036] Step 3.3: node feature update;

[0037] Introduce a multi-head attention mechanism, each head uses a different weight matrix W i Transform the features of neighbor nodes and then use the attention score e ij Normalized weighted sum is used to aggregate these transformed features; while calculating the node feature update, the F updated in step 3.2 is distance Incorporated into the calculation process as edge weight; the new feature h of node i i It is constructed by connecting the results of all head outputs; the calculation formula for node feature update is as follows:

[0038]

[0039] Among them, softmax(e ij ) represents the attention score e of all neighbors j of each node i ij Normalize, Fdistance (i, j) provides a dynamic weight based on feature differences; W i is the weight matrix used to transform the feature h of neighbor node j j ;σ is the ReLU activation function; Indicates that this layer uses K independent attention heads for feature aggregation, and each head uses a different weight matrix W k ; The results of each head aggregation are concatenated on the feature dimension to obtain the updated node feature h i ;

[0040] Step 3.4: Node feature propagation;

[0041] Using the updated h obtained from step 3.3 i As the new input feature of each node, the node feature propagation formula of each layer is as follows:

[0042] H (l+1) =σ(AH (l) W (l) )

[0043] Among them, H (l) is the feature set of all nodes in the lth layer after update in step 3.3 [h 1 ,h 2 ,…,h n ],W (l) is the weight matrix of the lth layer, σ is the ReLU activation function;

[0044] Step 3.5: Build GCN;

[0045] Each layer of GCN updates the node features through the node feature propagation formula in step 3.4 to capture the spatial dependencies between nodes; each GCN layer includes a graph convolution layer, a batch normalization layer, and a ReLU activation layer; among them, the graph convolution layer is used to aggregate the features of adjacent nodes, the batch normalization layer is used to normalize the layer output, thereby accelerating the training process and improving the generalization ability of the model, and the ReLU activation layer introduces nonlinearity to enhance the expressiveness of the model.

[0046] Furthermore, the temporal convolutional neural network TCN is implemented by stacking multiple temporal modules, and residual connections are used between the multiple temporal modules. Each temporal module includes a hybrid dilated convolution, a batch normalization layer, a random inactivation layer and a ReLU activation layer; the dilation coefficient of the hybrid dilated convolution is set to a periodic sequence of [1, 2, 3] to increase the receptive field of the network.

[0047] Furthermore, the specific process of fusing the graph convolutional neural network GCN and the temporal convolutional neural network TCN is as follows:

[0048] Step 4.1: Model fusion; Use the GCN constructed in step 3.5 to process the spatial relationship in the human body and extract the node features at each time step; then, use the output of GCN as the input of TCN and use TCN to process the dynamic changes of key points over time;

[0049] Step 4.2: Loss fusion: When training the model, back propagation and optimizer parameter update are performed according to the total loss function; the specific method is as follows:

[0050] Step 4.2.1: Calculate the mean square error (MSE) to measure the error between the key point positions predicted by the model and the actual data. The calculation formula is as follows:

[0051]

[0052] in, is the coordinate of the key point predicted by the model, J i is the corresponding real key point coordinate, N is the total number of key points;

[0053] Step 4.3.2: Calculate the temporal continuity loss to penalize the pose changes between adjacent frames in the time series. The calculation formula is as follows:

[0054]

[0055] Where T is the total number of frames, and are the predicted key point coordinates at time t and t+1, J t and J t+1 are the real key point coordinates at time t and t+1 respectively.

[0056] Step 4.3.3: Calculate the total loss; combine the two loss functions of step 4.3.1 and step 4.3.2 to form the final loss function, taking into account both spatial accuracy and temporal continuity. The total loss calculation formula is as follows:

[0057] Total Loss =λ 1 MSE Loss +λ 2 Temporal Consistency Loss

[0058] Among them, λ 1 and λ 2 is the weight used to balance the two loss components.

[0059] Furthermore, in the model training, data from the public dataset Human3.6M in the field of human pose estimation is used, and the open source framework PyTorch is used for neural network training; data enhancement is performed during training, and the occlusion operation of step 1.2 is integrated into the training data generation pipeline. Diverse training samples are generated through multiple random occlusions. The model encounters different occlusion situations in each iteration, thereby learning how to rely on visible key points to accurately estimate 3D poses. SGD is used as the optimizer, the learning rate is set to 0.005, the dimension of the attention vector is set to 128, the number of attention heads is set to 10, and the epoch is set to 100.

[0060] The beneficial effects of adopting the above technical solution are: the single-person 3D posture estimation method based on monocular video in occluded scenes provided by the present invention groups 2D key points based on prior knowledge of the human body, and performs data enhancement in the training stage by randomly shielding the 2D key point groups to simulate the human posture data in occluded scenes. This method not only effectively reduces the dependence on complex hardware, scene settings and manual annotation, but also enables the data set to be rapidly expanded by flexibly generating a variety of occlusion situations, thereby greatly improving the efficiency of data set construction; and implements the GCN anti-occlusion method based on directed graph weight adjustment. Node-level multi-head attention is introduced. By assigning different weights to different neighbor nodes, important neighbor information is emphasized, allowing the model to dynamically adjust the influence of neighbors on the target node; combined with the spatial modeling capabilities of GCN and the time series processing capabilities of TCN, the model can achieve better performance in human posture estimation tasks. BRIEF DESCRIPTION OF THE DRAWINGS

[0061] Figure 1 A flow chart of a method for single-person 3D posture estimation based on monocular video in an occlusion scene provided by an embodiment of the present invention;

[0062] Figure 2 17 key points of the human body in Human3.6M provided in the embodiment of the present invention;

[0063] Figure 3 A network architecture diagram of the integrated GCN-TCN provided in an embodiment of the present invention;

[0064] Figure 4 A comparison diagram of the effects after random occlusion enhancement provided in an embodiment of the present invention; wherein, Figure (4a) is the input image, Figure (4b) is the prediction result without using the random occlusion method, Figure (4c) is the prediction result using the random occlusion method, and Figure (4d) is the true value. DETAILED DESCRIPTION

[0065] The specific implementation of the present invention is further described in detail below in conjunction with the accompanying drawings and examples. The following examples are used to illustrate the present invention, but are not intended to limit the scope of the present invention.

[0066] like Figure 1 As shown, the method of this embodiment is described as follows.

[0067] Step 1: Occlusion condition simulation. 2D key points are grouped based on prior knowledge of the human body, and data enhancement is performed in the training phase by randomly occluding 2D key point groups to simulate human posture data in occlusion scenarios.

[0068] Step 1.1: 2D key point grouping. In the Human3.6M dataset, the 17 joint points of the human body are as follows: Figure 2 As shown, they are 0-root node, 1-right hip, 2-right knee, 3-right ankle, 4-left hip, 5-left knee, 6-left ankle, 7-base of spine, 8-chest, 9-neck, 10-head, 11-left shoulder, 12-left elbow, 13-left wrist, 14-right shoulder, 15-right elbow, 16-right wrist. First, based on the prior knowledge of the human body, the 2D key points of the human body are grouped. Common groups include feet, knees, legs, left limbs and right limbs. For example, the key point grouping G in the Human 3.6M dataset is as follows:

[0069] Both feet: G1 = [3,6],

[0070] Double knees: G2 = [2,3,5,6],

[0071] Legs: G3 = [0, 1, 2, 3, 4, 5, 6],

[0072] Left side: G4 = [1,2,3,14,15,16],

[0073] Right side: G5=[4,5,6,11,12,13].

[0074] Step 1.2: Random occlusion strategy setting. According to the occlusion characteristics of the application scenario, set the occlusion probability P for different groups G in step 1.1, and set P1 = 0.3, P2 = 0.3, P3 = 0.2, P4 = 0.1, and P5 = 0.1. For each group of key points, generate a random number. If the random number is less than the occlusion probability of the group, mask all 2D key points of the group and fill their values ​​with zeros.

[0075] Step 1.3: Data enhancement during training. Integrate the masking operation in step 1.2 into the training data generation pipeline, and generate diverse training samples through multiple random occlusion operations. Using the enhanced dataset for model training, the model will encounter different occlusion situations in each iteration. Through this diverse occlusion scene learning, the model will be able to infer how to rely on visible key points to accurately estimate 3D poses when some key points are missing.

[0076] Step 2: Directed graph construction. The directionality of human bones is incorporated into the graph construction process. Traditional undirected graphs cannot reflect the movement direction between joints. In directed graphs, a directed graph is constructed by adjusting the adjacency matrix.

[0077] Step 2.1: Define the adjacency matrix. The adjacency matrix is ​​used to represent the connectivity between key points of the human body. In an undirected graph, the connection is symmetric, that is, if there is a connection between node i and node j, then A ij =1 and A ji = 1. However, in a directed graph, the connection is unidirectional, the adjacency matrix is ​​no longer symmetric, and A ij =1 indicates a unidirectional connection from node i to node j. When constructing a directed graph, the connection direction in the adjacency matrix needs to be adjusted according to the natural movement order of the human skeleton. Usually, movement is transmitted from top to bottom or from the trunk to the limbs, so the following matrix is ​​constructed to reflect this movement hierarchy.

[0078]

[0079] Step 2.2: Define the basic feature matrix. The basic feature matrix is ​​the Euclidean distance between joints. It is used in conjunction with the adjacency matrix to define the connections between joints. The basic feature matrix also provides additional weights or features for these connections. The basic feature matrix is ​​constructed using the Euclidean distance between joints of the standard human body as the initial value. Define the Euclidean distance d between joints. ij , which represents the distance between node i and node j, and the calculation formula is as follows:

[0080]

[0081] Using a 17×17 basic feature matrix F distance To store the Euclidean distance between all joints, F distance It is expressed as follows:

[0082]

[0083] Step 3: Implement the GCN anti-occlusion method based on directed graph weight adjustment. Introduce node-level multi-head attention. By assigning different weights to different neighbor nodes, important neighbor information is emphasized, allowing the model to dynamically adjust the influence of neighbors on the target node.

[0084] Step 3.1.: Attention score calculation. First, calculate the attention score e ij , the attention score is obtained by combining the node features with the learnable parameter a T Perform a linear combination and apply the activation function σ to calculate it. The calculation formula is as follows:

[0085]

[0086] Among them, h i ||h j is the concatenation of the feature vectors of node i and node j, a T is a learnable weight vector used to parameterize the attention mechanism, σ is the ReLU activation function, and d k is a normalization factor related to the degree of the node.

[0087] Step 3.2: Update F distance For each pair of nodes i and j, F is updated based on the difference in the Euclidean distance between their feature vectors. distance . F distance The update formula is as follows:

[0088] F distance (i,j)=exp(-||h i -h j ||)

[0089] Among them, ||h i -h j || is the Euclidean distance between the features of node i and node j, which is converted into a weight value close to the interval [0,1] through an exponential function.

[0090] Step 3.3: Node feature update. Introduce a multi-head attention mechanism, each head uses a different weight matrix W i Transform the features of neighbor nodes and then use the attention score e ij The normalized weighted sum is used to aggregate these transformed features. While calculating the node feature update, the F in step 3.2 is distance As edge weights, they are integrated into the calculation process. The new feature h of node i i It is constructed by connecting the results of all head outputs. The calculation formula for node feature update is as follows:

[0091]

[0092] Among them, e ij is the attention score of neighbor node j to node i, softmax(e ij ) for each node i’s neighbors j’s e ij Normalize, F distance (i, j) provides a dynamic weight based on feature differences, W i is the weight matrix used to transform the feature h of neighbor node j j , σ is the ReLU activation function, Indicates that this layer uses K independent attention heads for feature aggregation, and each head uses a different weight matrix W k The results of each head aggregation are concatenated in the feature dimension to obtain the updated node feature h i .

[0093] Step 3.4: Node feature propagation. Using the h obtained from step 3.3 i As the new input feature of each node, the node feature propagation formula of each layer is as follows:

[0094] H (l+1) =σ(AH (l) W (l) )

[0095] Among them, H (l) is the feature set of all nodes in the lth layer after update in step 3.3 [h 1 ,h 2 ,…,h n ],W (l) is the weight matrix of the lth layer, and σ is the ReLU activation function.

[0096] Step 3.5: Build GCN. Each layer of GCN updates the node features through the node feature propagation formula of the node in step 3.4, thereby capturing the spatial dependencies between nodes. Each GCN layer contains a graph convolution layer, a batch normalization layer, and a ReLU activation layer. Among them, the graph convolution layer is used to aggregate the features of adjacent nodes, the batch normalization layer is used to normalize the layer output, thereby accelerating the training process and improving the generalization ability of the model, and the ReLU activation layer introduces nonlinearity to enhance the expression ability of the model.

[0097] Step 4: Fusion of GCN and TCN.

[0098] Step 4.1: Construct TCN. TCN is implemented by stacking multiple time series modules, and residual connections are used between multiple time series modules. Each time series module includes a mixed dilated convolution, a batch normalization layer, a random dropout layer, and a ReLU activation layer; the dilation coefficient of the mixed dilated convolution is set to a periodic sequence of [1, 2, 3] to increase the receptive field of the network.

[0099] Step 4.2: Model fusion. Use the GCN constructed in step 3 to process the spatial relationship in the human body and extract the node features at each time step. Then, use the output of GCN as the input of TCN and use TCN to process the dynamic changes of key points over time. The network architecture diagram of the fusion of GCN-TCN is as follows: Figure 3 shown.

[0100] Step 4.3: Loss fusion: When training the model, back propagation and optimizer parameter update are performed according to the total loss function.

[0101] Step 4.3.1: Calculate the mean square error (MSE). It is used to measure the error between the key point positions predicted by the model and the actual data. The calculation formula is as follows:

[0102]

[0103] in, is the coordinate of the key point predicted by the model, J i are the corresponding real key point coordinates, and N is the total number of key points, which is 17.

[0104] Step 4.3.2: Calculate the temporal continuity loss. To ensure that the predicted posture is physically reasonable and smooth between consecutive frames, the temporal continuity loss is introduced to penalize the posture changes between adjacent frames in the time series. The calculation formula is as follows:

[0105]

[0106] Where T is the total number of frames, and are the predicted key point coordinates at time t and t+1, J t and J t+1 are the real key point coordinates at time t and t+1 respectively.

[0107] Step 4.3.3: Calculate the total loss. Combine the two loss functions of step 4.3.1 and step 4.3.2 to form the final loss function, taking into account both spatial accuracy and temporal continuity. The calculation formula is as follows:

[0108] Total Loss =λ 1 MSE Loss +λ 2 Temporal Consistency Loss

[0109] Among them, λ 1 and λ 2 is the weight used to balance the two loss components.

[0110] Step 5: Model training.

[0111] In terms of data, this embodiment uses data from the public dataset Human3.6M in the field of human pose estimation for training. In terms of training, the open source framework PyTorch is used for neural network training. Data enhancement is performed during training, and the occlusion operation in step 1.2 is integrated into the training data generation pipeline. Diverse training samples are generated through multiple random occlusions. The model encounters different occlusion situations in each iteration, so it learns how to accurately estimate 3D poses based on visible key points. The effect of random occlusion enhancement is compared. Figure 4 As shown, Figure (4a) is the input image, Figure (4b) is the prediction result without using the random occlusion method, Figure (4c) is the prediction result using the random occlusion method, and Figure (4d) is the true value. SGD is used as the optimizer, the learning rate is set to 0.005, the dimension of the attention vector is set to 128, the number of attention heads is set to 10, and the epoch is set to 100.

[0112] This embodiment uses the public dataset Human3.6M in the field of human posture estimation for evaluation and comparison. As shown in Table 1, the method of this embodiment achieves higher accuracy than similar 3D human posture estimation methods. Figure 2 The prediction effect under the condition of severe self-occlusion (SittingDown action) is further demonstrated. Compared with other methods, the present invention can maintain higher accuracy.

[0113] Table 1 Performance comparison of the present invention and other methods on the Human3.6M dataset

[0114]

[0115] In summary, the present invention proposes a single-person 3D pose estimation method based on monocular video in occlusion scenarios. There is no need to manually construct a human pose estimation dataset with occlusion conditions. By randomly shielding 2D key points, a variety of occlusion conditions can be flexibly generated in the training stage. The fused GCN and TCN models can still maintain high accuracy under occlusion conditions, which is conducive to the implementation of human pose estimation technology in practical applications.

[0116] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the above embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the above embodiments, or replace some or all of the technical features therein with equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope defined by the claims of the present invention.

Claims

1. A single-person 3D pose estimation method based on monocular video in occluded scenes, characterized by: The 2D key points are grouped using prior knowledge of the human body, and data enhancement is performed by randomly occluding some 2D key point groups during the model training stage to simulate human posture data in occluded scenarios. A 3D human posture estimation network architecture that integrates graph convolutional neural network (GCN) and temporal convolutional neural network (TCN) is designed, in which GCN introduces a directed graph weight adjustment module for human posture estimation in the spatial dimension. TCN captures dynamic changes between frames in the temporal dimension based on mixed dilated convolution. The output of GCN is used as the input of TCN to achieve joint modeling of spatial and temporal information.

2. The method for estimating a single person's 3D posture based on monocular video in an occluded scene according to claim 1, characterized in that: The simulation of human body posture data in the occlusion scene specifically includes the following steps: Step 1.1: 2D key point grouping; In the Human3.6M dataset, the 17 joints of the human body are 0-root node, 1-right hip, 2-right knee, 3-right ankle, 4-left hip, 5-left knee, 6-left ankle, 7-spine base, 8-chest, 9-neck, 10-head, 11-left shoulder, 12-left elbow, 13-left wrist, 14-right shoulder, 15-right elbow, 16-right wrist; First, based on the prior knowledge of the human body, the 2D key points of the human body are grouped, including feet, knees, legs, left limbs and right limbs; the key point grouping G in the Human 3.6M dataset is as follows: Both feet: G1 = [3,6], Double knees: G2 = [2,3,5,6], Legs: G3 = [0, 1, 2, 3, 4, 5, 6], Left side: G4 = [1,2,3,14,15,16], Right side: G5 = [4,5,6,11,12,13]; Step 1.2: Set a random occlusion strategy; according to the occlusion characteristics of the application scenario, set the occlusion probability P for different groups G in step 1.1, and the occlusion probabilities corresponding to the five groups are set to P1=0.3, P2=0.3, P3=0.2, P4=0.1, and P5=0.1 respectively; for each group of key points, generate a random number. If the random number is less than the occlusion probability of the group, occlude all 2D key points of the group and fill their values ​​with zeros.

3. The method for estimating a single person's 3D posture based on monocular video in an occluded scene according to claim 2, characterized in that: In the construction of the directed graph, the directionality of the human skeleton is incorporated into the graph construction process, as follows: Step 2.1: Define the adjacency matrix to represent the connectivity between key points of the human body; in a directed graph, the connection is unidirectional, the adjacency matrix is ​​asymmetric, A ij =1 indicates a unidirectional connection from node i to node j; when constructing a directed graph, the connection direction in the adjacency matrix is ​​adjusted according to the natural movement order of the human skeleton; Step 2.2: Define the basic feature matrix; The basic feature matrix is ​​the Euclidean distance between joints. It is used in conjunction with the adjacency matrix to define the connections between joint points, while providing additional weights or features for these connections through the basic feature matrix. The Euclidean distance between the joints of the standard human body is used as the initial value to construct the basic feature matrix; Define the Euclidean distance d between joints ij , represents the distance between node i and node j, and the calculation formula is as follows: Using a 17×17 basic feature matrix To store the Euclidean distances between all joints.

4. The method for estimating a single person's 3D posture based on monocular video in an occluded scene according to claim 3, characterized in that: In step 2.1, the natural movement of the human skeleton is transmitted from top to bottom or from the trunk to the limbs, so the following matrix is ​​constructed to reflect this movement hierarchy:

5. The method for estimating a single person's 3D posture based on monocular video in an occluded scene according to claim 4, characterized in that: The directed graph weight adjustment module implements the GCN anti-occlusion method based on directed graph weight adjustment, introduces node-level multi-head attention, and emphasizes important neighbor information by assigning different weights to different neighbor nodes, so that the model can dynamically adjust the influence of neighbors on the target node; the specific method is as follows: Step 3.1: Attention score calculation; Attention score ij By combining the node features with the learnable parameters a T Perform a linear combination and apply the activation function σ to calculate it. The calculation formula is as follows: Among them, e ij is the attention score of neighbor node j to node i, h i ||h j is the concatenation of the feature vectors of node i and node j; a T is a learnable weight vector used to parameterize the attention mechanism; σ is the ReLU activation function; d k is a normalization factor related to the degree of the node; Step 3.2: Update the basic feature matrix F distance ; For each pair of nodes i and j, F is updated based on the difference in the Euclidean distance between their feature vectors. distance ; F distance The update formula is as follows: F distance (i,j)=exp(-∥h i -h j ∥) Among them, ∥h i -h j ∥ is the Euclidean distance between the features of node i and node j, which is converted into a weight value close to the interval [0,1] through the exponential function; Step 3.3: node feature update; Introduce a multi-head attention mechanism, each head uses a different weight matrix W i Transform the features of neighbor nodes and then use the attention score e ij Normalized weighted sum is used to aggregate these transformed features; while calculating the node feature update, the F updated in step 3.2 is distance Incorporated into the calculation process as edge weight; the new feature h of node i i It is constructed by connecting the results of all head outputs; the calculation formula for node feature update is as follows: Among them, softmax(e ij ) represents the attention score e of all neighbors j of each node i ij Normalize, F distance (i, j) provides a dynamic weight based on feature differences; W i is the weight matrix used to transform the feature h of neighbor node j j ;σ is the ReLU activation function; Indicates that this layer uses K independent attention heads for feature aggregation, and each head uses a different weight matrix W k ; The results of each head aggregation will be spliced ​​in the feature dimension to obtain the updated node feature h i ; Step 3.4: Node feature propagation; Using the updated h obtained from step 3.3 i As the new input feature of each node, the node feature propagation formula of each layer is as follows: H (l+1) =σ(AH (l) W (l) ) Among them, H (l) is the feature set of all nodes in the lth layer after update in step 3.3 [h1,h2,…,h n ],W (l) is the weight matrix of the lth layer, σ is the ReLU activation function; Step 3.5: Build GCN; Each layer of GCN updates the node features through the node feature propagation formula in step 3.4 to capture the spatial dependencies between nodes; each GCN layer includes a graph convolution layer, a batch normalization layer, and a ReLU activation layer; among them, the graph convolution layer is used to aggregate the features of adjacent nodes, the batch normalization layer is used to normalize the layer output, thereby accelerating the training process and improving the generalization ability of the model, and the ReLU activation layer introduces nonlinearity to enhance the expressiveness of the model.

6. The method for estimating a single person's 3D posture based on monocular video in an occluded scene according to claim 5, characterized in that: The temporal convolutional neural network (TCN) is implemented by stacking multiple temporal modules. Residual connections are used between the multiple temporal modules. Each temporal module includes a mixed dilated convolution, a batch normalization layer, a random loss layer, and a ReLU activation layer. The dilation coefficient of the mixed dilated convolution is set to a periodic sequence of [1, 2, 3] to increase the receptive field of the network.

7. The method for estimating a single person's 3D posture based on monocular video in an occluded scene according to claim 6, characterized in that: The specific process of fusing the graph convolutional neural network GCN and the temporal convolutional neural network TCN is as follows: Step 4.1: Model fusion; Use the GCN constructed in step 3.5 to process the spatial relationship in the human body and extract the node features at each time step; then, use the output of GCN as the input of TCN and use TCN to process the dynamic changes of key points over time; Step 4.2: Loss fusion: When training the model, back propagation and optimizer parameter update are performed according to the total loss function; the specific method is as follows: Step 4.2.1: Calculate the mean square error (MSE) to measure the error between the key point positions predicted by the model and the actual data. The calculation formula is as follows: in, is the coordinate of the key point predicted by the model, J i is the corresponding real key point coordinate, N is the total number of key points; Step 4.3.2: Calculate the temporal continuity loss to penalize the pose changes between adjacent frames in the time series. The calculation formula is as follows: Where T is the total number of frames, and are the predicted key point coordinates at time t and t+1, J t and J t+1 are the real key point coordinates at time t and t+1 respectively; Step 4.3.3: Calculate the total loss; combine the two loss functions of step 4.3.1 and step 4.3.2 to form the final loss function, taking into account both spatial accuracy and temporal continuity. The total loss calculation formula is as follows: Total Loss =λ1·MSE Loss +λ2·Temporal Consistency Loss Among them, λ1 and λ2 are weights used to balance the two loss components.

8. The method for estimating a single person's 3D posture based on monocular video in an occluded scene according to claim 7, characterized in that: In the model training, data from the public dataset Human3.6M in the field of human pose estimation is used, and the open source framework PyTorch is used for neural network training; data enhancement is performed during training, and the occlusion operation in step 1.2 is integrated into the training data generation pipeline. Diverse training samples are generated through multiple random occlusions. The model encounters different occlusion situations in each iteration, thereby learning how to rely on visible key points to accurately estimate 3D poses; SGD is used as the optimizer, the learning rate is set to 0.005, the dimension of the attention vector is set to 128, the number of attention heads is set to 10, and the epoch is set to 100.

Images