Single-view occluded human motion reconstruction method based on self-supervised spatiotemporal motion prior

Through self-supervised learning and space-time motion prior encoder, the problem of single-view occlusion of human motion reconstruction is solved, real-time three-dimensional motion reconstruction under a single RGB camera is realized, and the accuracy of reconstruction and the convenience of deployment is improved.

CN114926594BActive Publication Date: 2024-11-12SOUTHEAST UNIV
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202210684494.4
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2022-06-17
Publication Date
2024-11-12
Estimated Expiration
2042-06-17

AI Technical Summary

Technical Problem

The existing single-view method is difficult to realize the reconstruction of human movement in occlusion. The lack of sufficient occlusion of three-dimensional human movement data and the lack of depth of single-view images leads to underdetermined information, and it is difficult for network models to return to reliable and accurate three-dimensional motion.

Method used

Using self-supervised method, the space-time prior knowledge of shaking human motion is learned by synthesizing shaking human motion data, and a space-time prior encoder based on expansion convolution and Transformer is constructed to assist three-dimensional motion reconstruction network training to realize real-time single-view shaking human motion reconstruction.

Benefits of technology

Real-time single-view angle occluding the human body's three-dimensional motion reconstruction under a single RGB camera is realized, reducing the ambiguity of occluding the two-dimensional human body's motion, and improving the accuracy of reconstruction and the convenience of deployment.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN114926594B_ABST
    Figure CN114926594B_ABST
Patent Text Reader

Abstract

The present invention provides a method for reconstructing occluded human motion sequences based on self-supervised spatiotemporal motion priors, comprising the following steps: S1. Human motion synthesis and representation; S2. Construction of spatiotemporal prior networks for occluded human bodies; S3. Training of spatiotemporal prior networks for occluded human bodies; S4. Construction of three-dimensional motion reconstruction networks; S5. Training of three-dimensional motion reconstruction networks; S6. Global position estimation; S7. Real-time single-view occluded human motion reconstruction. The present invention can quickly synthesize a large amount of occluded data without affecting the generalization ability of the model on real data, thus solving the problem of the existing methods' strong dependence on real occluded human body data.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the fields of computer vision and three-dimensional vision, and in particular to a single-view occluded human body motion reconstruction method based on self-supervised spatiotemporal motion prior. Technical Background

[0002] Human motion reconstruction plays a very important role in applications such as holographic communication, behavior analysis, and motion capture. With the continuous development of related technologies such as artificial intelligence, single-view human motion reconstruction based on a single RGB camera has a wide market demand due to its low cost and convenient deployment. However, existing single-view methods are difficult to achieve human motion reconstruction under occlusion. Single-view occluded human motion reconstruction has become a problem that needs to be solved urgently. At present, human motion reconstruction solutions for occlusion problems face two major difficulties: first, there is a lack of sufficient occluded three-dimensional human motion data, and it is difficult to train a model with good generalization performance through deep learning; second, the lack of depth in single-view images and the lack of information caused by occlusion lead to serious ambiguity, and it is difficult for the network model to regress reliable and accurate three-dimensional motion. Therefore, it is a practical and challenging problem to use self-supervised methods to learn the prior knowledge of occluded human spatiotemporal motion using synthetic occluded human motion data, model the spatiotemporal characteristics at the joint level, and achieve occluded human motion reconstruction. Summary of the invention

[0003] Purpose of the invention: The present invention proposes a method for reconstructing a single-view occluded human body motion sequence based on self-supervised spatiotemporal motion prior. The occluded motion sequence is synthesized using real two-dimensional human body data to construct a training data set. By constructing a human body motion spatiotemporal prior encoder based on dilated convolution and Transformer, the spatiotemporal features of the local joint point level and the global motion level of the occluded human body motion are learned in the synthetic two-dimensional training data. The trained human body motion spatiotemporal prior encoder is then used to assist in the training of the three-dimensional motion reconstruction network. During real-time single-view occluded human body motion reconstruction, the joint point map of the two-dimensional occluded human body motion is obtained from the single-view occluded human body video, and the joint point map of the two-dimensional occluded human body motion is sent to the trained network for forward reasoning and global position estimation, thereby achieving real-time three-dimensional motion reconstruction of the occluded human body.

[0004] Technical solution: The method for reconstructing occluded human motion sequences based on self-supervised spatiotemporal motion priors described in the present invention comprises the following steps:

[0005] S1. Human motion synthesis and representation: In order to overcome the high cost of producing occluded human data sets, the present invention proposes a two-dimensional occluded human motion joint point graph representation and a human occlusion data generation method to quickly synthesize a large amount of occlusion data without affecting the generalization ability of the model on real data. By using instance segmentation in network videos to obtain occluders, and randomly overlaying the occluders on the pictures in the human data set without occlusion to achieve synthetic occlusion, a two-dimensional occluded human motion joint point graph is generated on the two-dimensional complete human motion joint point graph for network training.

[0006] S2. Construction of spatiotemporal prior network for occluded human body: Occluded two-dimensional human body joints are highly ambiguous and difficult to be directly used to estimate accurate three-dimensional complete human body motion. The present invention constructs an encoder including a spatiotemporal feature extraction network at the local joint level and a spatiotemporal feature extraction network at the global motion level to extract spatiotemporal features at different levels respectively. In addition, an encoder composed of a fully connected network is constructed to regress the joint point graph of two-dimensional complete human body motion from the code estimated by the encoder.

[0007] S3. Training of the spatiotemporal prior network for occluded human bodies: The synthesized 2D occluded human body motion joint point graph is used as input, and the corresponding 2D complete human body motion joint point graph without occlusion is used as supervision to train the spatiotemporal prior network for occluded human bodies constructed in S2 in a self-supervised manner. This neural network is trained in a self-supervised manner until convergence, and the prior knowledge of occluded 2D human body motion is learned.

[0008] S4. Construction of 3D motion reconstruction network: Construct the same network as the spatiotemporal prior network encoder for occluded human body in S2 as the encoder, with a fully connected layer and two Transformer modules as the decoder, which is used to estimate the 3D complete human body motion map from the joint point map of the 2D occluded human body motion.

[0009] S5. 3D motion reconstruction network training: The parameters of the occluded human body spatiotemporal prior network encoder trained in step S3 are used as pre-training parameters and assigned to the encoder of the three-dimensional motion reconstruction network constructed in S4, so that the 3D motion reconstruction network obtains the prior knowledge of the occluded 2D human body motion. The synthesized 2D occluded human body motion joint point map is further used as input and the 3D complete human body motion map is used as supervision for network training. During the training process, the encoder parameters are also fine-tuned. Training is performed until convergence.

[0010] S6. Global position estimation: A 3D human motion sequence is sampled from the 3D complete human motion map estimated in S4. Since the sampled 3D human motion sequence is in a local coordinate system, in order to obtain a 3D human motion sequence with an absolute position, the present invention uses the joint point positions of the 3D human sequence skeleton and the input occluded 2D joint point positions to construct a least squares function to estimate the global translation. The estimated translation is applied to the 3D human motion sequence.

[0011] S7. Real-time single-view occluded human motion reconstruction: After network training is completed. Use a single RGB camera to build a single-view motion capture system. Input a human motion video sequence captured by a single-view camera, use the existing open source two-dimensional joint point detection method to detect the joint point position and its confidence for each frame, and obtain a two-dimensional joint point map of occluded human motion. Estimate the three-dimensional complete human motion map through the three-dimensional motion reconstruction network. And use the method in step S6 to obtain a three-dimensional human motion sequence with absolute position. Realize single-view occluded human three-dimensional motion reconstruction. The global human motion sequence can be further skinned to obtain a deformable human network model.

[0012] Furthermore, the specific method of step S1 includes:

[0013] S11. Random occluder acquisition: Mask-RCNN is used to obtain occluder masks and RGB images in online videos using instance segmentation to simulate occlusion.

[0014] S12. Acquisition of joint point graph of 2D complete human motion: Extract the 2D human motion sequence from the human data set, where the 2D human motion of each frame is represented by the coordinates of K human joint points. For the 2D human motion sequence with K joint points in F frames, in order to facilitate network calculation, first subtract the root node coordinate (x) from all the related node coordinates (x, y). root ,y root ) is then divided by the skeleton bounding box size w×h to achieve normalization, and the normalized point coordinates are Stored in a two-dimensional joint point graph I 2d ∈R F×K×2 Each row in the figure represents the normalized position of all nodes in a certain frame.

[0015] S13. Acquisition of joint point graph of 2D occluded human motion: Since the joint point graph representation of human motion is not affected by the image appearance, the occluded human motion data can be obtained by adding occluders to the non-occluded human image. Calculate the intersection-and-union ratio between the skeleton bounding box and the occluder obtained in S11, and make the intersection-and-union ratio between X1 and X2 to achieve occlusion of different occlusion ratios. For the mask image of the occluder, by setting the value of the joint points inside the mask image to 0 and the value of the joint points outside the mask image to 1, a confidence map I will be generated. c ∈R F×K×1 Multiply the confidence map and the two-dimensional joint point map to get the two-dimensional joint point map of the occluded human body motion This representation method can quickly synthesize a large amount of occlusion data without affecting the model's generalization ability on real data.

[0016] S14. Acquisition of 3D complete human motion graph: Extract 3D human motion sequences from existing human data sets, where each frame of human 3D motion is represented by a 3D human model with a skeleton skin. For a 3D human motion sequence with N joints in F frames, the skeleton posture of the 3D human model is represented by joint point rotation, and the human mesh is deformed by the joint point rotation, which is stored in the 3D complete human motion graph I. 3d ∈R F×N×6 middle.

[0017] Furthermore, the specific method of step S2 is:

[0018] S21.Since considering temporal and spatial relationships separately at different stages will lead to information loss, the encoder constructed contains two modules: a local joint-level spatiotemporal feature extraction network and a global motion-level spatiotemporal feature extraction network to extract spatiotemporal features at different levels. The local spatiotemporal relationship module is used to model local spatiotemporal features. It contains four convolutional layers, of which the first three convolutional layers are convolutional layers with expansion coefficients of 1, 2, and 5, respectively. The outputs of the three expansion convolutional layers are concatenated and then fused with spatiotemporal features through the last convolutional layer.

[0019] S22. Due to the limited performance of the convolutional layer in terms of temporal continuity, a global motion-level spatiotemporal feature extraction network is further constructed to model global spatiotemporal features. The global motion-level spatiotemporal feature extraction network includes a global spatial relationship module and a global temporal relationship module, and adopts a Transformer network structure. Spatial embedding is added to the output of the first module, and temporal embedding is added to the output of the second module.

[0020] S23. Construct an encoder consisting of a fully connected network to regress the joint point map of the two-dimensional complete human motion from the encoding estimated by the encoder.

[0021] Furthermore, the specific method of step S3 is:

[0022] S31. In order to enhance the generalization performance of the network, data augmentation is performed on the training data. Specific strategies include: 1) Mirror flipping. Since the human body has left-right symmetry, the human body data is multiplied by exchanging the rotation angles of symmetrical joints; 2) Sampling at different rates. A new human body motion sequence is obtained by sampling the original human body motion sequence at different sampling rates; 3) Reverse sampling. The order of the original human body motion sequence is reversed as a new motion sequence.

[0023] S32. Take the synthesized two-dimensional occluded human motion joint point graph as input, and output the two-dimensional complete human motion joint point graph through the occluded human spatiotemporal prior network. The two-dimensional complete human motion joint point graph output by the occluded human spatiotemporal prior network is supervised by the true value of the two-dimensional complete human motion joint point graph. L1 loss is used as a constraint, and the formula is as follows:

[0024]

[0025] Among them I 2d and They represent the reconstructed joint point graph of the two-dimensional complete human motion and the true value of the joint point graph of the two-dimensional complete human motion, ‖·‖1 represents the one-norm, and L 2d Represents the output 2D full human motion joint point graph and the L1 loss of the corresponding ground truth.

[0026] In addition, the smoothness term is further used to constrain the joint point graph of the output two-dimensional complete human motion:

[0027]

[0028] in and They represent the two-dimensional joint points estimated at the current moment and the next moment, that is, the parameters of the t-th row and t+1-th row of the joint point diagram of the two-dimensional complete human motion. smooth,2d Smoothness constraints on the joint graph representing the output 2D full human motion.

[0029] Finally, the training loss of the occluded human spatiotemporal prior network is:

[0030] L self =ω1L 2d +ω2L smooth,2d

[0031] Where ω1 and ω2 are weights.

[0032] Further, the specific method of step S4 is:

[0033] S41. Construct the same network as the spatiotemporal prior network encoder for occluded human body in S2 as the encoder, and use a fully connected layer and two Transformer modules for estimating temporal and spatial relationships respectively as the decoder to estimate the 3D complete human body motion map from the 2D occluded human body motion joint point map.

[0034] Further, the specific method of step S5 is:

[0035] S51. The parameters of the spatiotemporal prior network encoder of the occluded human body trained in step S3 are used as pre-training parameters, and assigned to the encoder of the three-dimensional motion reconstruction network constructed in S4, so that the three-dimensional motion reconstruction network obtains the prior knowledge of the occluded two-dimensional human body motion.

[0036] S52. Take the synthesized two-dimensional occluded human motion joint point map as input, and output a three-dimensional complete human motion map after encoding and decoding by the three-dimensional motion reconstruction network. The output three-dimensional complete human motion map is constrained using the reconstructed L2 loss, and the formula is as follows:

[0037]

[0038] Among them I 3d ,V 3d ,J 3d are the reconstructed complete 3D human motion graph, vertex coordinates and joint coordinates, β is the parameter representing the human body shape, the superscript ^ represents the true value of each parameter obtained from the data set, and L rec represents the L2 loss of reconstruction.

[0039] Furthermore, an additional regularization constraint is used to prevent abnormal human body shapes:

[0040] L reg =‖β‖2

[0041] In addition, the output 3D complete human motion graph is further constrained by a smoothness term:

[0042]

[0043] where θ is the value from I 3d The human motion parameters sampled in , M(θ) is the motion feature obtained by motion prior encoding, L smooth,3d Represents the smoothness term constraint on the output 3D complete human motion map.

[0044] Finally, the training loss of the 3D motion reconstruction network is:

[0045] L 3dbranch =L rec +ω3L reg +ω4Lsmooth,3d

[0046] Where ω3 and ω4 are weights. During the training process, the encoder parameters are fine-tuned at the same time. The training is continued until convergence.

[0047] Further, the specific method of step S6 is:

[0048] S61. Sample a 3D human motion sequence from the 3D complete human motion map estimated in S4. Since the sampled 3D human motion sequence is in a local coordinate system, in order to obtain a 3D human motion sequence with an absolute position, the joint point positions of the 3D human sequence skeleton and the input occluded 2D joint point positions are used to construct a least squares function to estimate the global translation. The estimated translation is applied to the 3D human motion sequence. The following least squares function is constructed:

[0049]

[0050] Where K is the camera internal parameter, P is the coordinates of the two-dimensional joint points of the human body sampled and restored in the joint point map of the two-dimensional occluded human body motion, ω c is the confidence of the corresponding joint point, It is the translation amount of the skeleton-skinned 3D human body model. By adding the translation amount to the skeleton-skinned 3D human body model, the 3D motion of the human body with absolute position can be obtained.

[0051] Further, the specific method of step S7 is:

[0052] S71. Use a single camera to capture video. Use a tripod to fix the camera and capture the obstructed human body.

[0053] S72. Use the existing open source two-dimensional joint point detection method to detect the joint point position and its confidence for each frame of the video sequence collected in step S71. Store the joint point position as a two-dimensional joint point map according to the method in S13. Binarize the confidence, assign 1 to values ​​greater than the threshold X, and assign 0 to values ​​less than the threshold X, and store the result in a confidence map of F×N×1. Multiply the confidence map and the two-dimensional joint point map to obtain a two-dimensional joint point map that occludes human body motion.

[0054] S73. Input a two-dimensional occluded human body motion joint point diagram, and output a reconstructed three-dimensional complete human body motion diagram through the occluded human body spatiotemporal prior network and the three-dimensional motion reconstruction network.

[0055] S74. Obtain the global position of each posture through S61. Finally, the three-dimensional motion of the human body with absolute position is obtained by adding translation parameters to the three-dimensional human body model with skeleton skin. The global human body motion sequence can be further skinned to obtain a deformable human body network model.

[0056] Beneficial effects: Compared with the prior art, the beneficial effects of the present invention are as follows: 1. A joint point graph representation of two-dimensional occluded human body motion and a method for generating human body occlusion data are proposed, which can quickly synthesize a large amount of occluded data without affecting the generalization ability of the model on real data, solving the strong dependence of existing methods on real occluded human body data. 2. The two-dimensional occluded human body spatiotemporal motion prior is trained by using the self-supervised training strategy of synthetic data and models, which improves the accuracy of occluded motion reconstruction. 3. A local joint point spatiotemporal feature extraction network based on dilated convolution and a global motion level spatiotemporal feature extraction network based on Transformer are proposed to effectively extract spatiotemporal features at different levels, reduce the ambiguity of occluded two-dimensional human body motion, and improve the accuracy of three-dimensional human body motion reconstruction. 4. This method only relies on a single RGB camera to achieve real-time occluded human body motion reconstruction, which is easy to deploy and low cost. BRIEF DESCRIPTION OF THE DRAWINGS

[0057] Figure 1 It is a framework diagram for training priors of occlusion space-time motion;

[0058] Figure 2 This is a diagram of the training framework of the occluded human motion reconstruction network;

[0059] Figure 3 It is a flowchart of occluded human motion reconstruction;

[0060] Figure 4 is a schematic diagram of an occluder, wherein the previous row is an occluder diagram, and the next row is an occluder mask diagram;

[0061] Figure 5 is a schematic diagram of synthetic occlusion, where the upper row is the synthetic occlusion map and the lower row is the corresponding original image;

[0062] Figure 6 2. It is a schematic diagram of synthesizing occluded two-dimensional motion sequences, wherein the left figure is a schematic diagram of synthesized occlusion, the middle figure is a synthetic occlusion mask diagram, and the right figure is a two-dimensional occluded human body motion joint point diagram;

[0063] Figure 7 It is a schematic diagram of a 3D human body motion diagram, where the left picture is a complete 3D human body motion diagram, and the right picture is a reconstruction result diagram;

[0064] Figure 8 These are the results of the reconstructed 3D motion after skinning, where the left column is the original image, the middle column is the skinned result rendering, and the right column is the rendering result from different perspectives. DETAILED DESCRIPTION

[0065] The present invention is further described in detail below with reference to the accompanying drawings. The implementation process of the method for reconstructing occluded human motion sequences based on self-supervised spatiotemporal motion priors described in the present invention is as follows:

[0066] S11. Random occluder acquisition: Figure 4 As shown in Figure 1, Mask-RCNN is used to obtain occluder masks and RGB images in online videos using instance segmentation to simulate occlusion.

[0067] S12. Extract a human 2D motion sequence from the human data set, where the 2D motion of each frame is represented by the coordinates of 24 human joints. For a 24-frame human 2D motion sequence with 24 joints, in order to facilitate network calculation, first subtract the root node coordinate (x) from all the node coordinates (x, y). root ,y root ) is then divided by the skeleton bounding box size w×h to achieve normalization, and the normalized point coordinates are Stored in a two-dimensional joint point graph I 2d ∈R 24×24×2 Each row in the figure represents the normalized position of all nodes in a certain frame.

[0068] S13. Acquisition of joint point graphs for 2D occluded human motion: Since the joint point graph representation of human motion is not affected by the image appearance, e.g. Figure 5 As shown in , by adding an occluder to a non-occluded human image, the occluded human motion data can be obtained. Figure 6 As shown, the intersection-and-union ratio between the skeleton bounding box and the occluder obtained in S11 is calculated, and the intersection-and-union ratio is made between X1 and X2 to achieve occlusion of different occlusion ratios. For the mask image of the occluder, a confidence map I is generated by stipulating that the value of the joint point inside the mask image is 0 and the value of the joint point outside the mask image is 1. c ∈R 24×24×1 Multiply the confidence map and the two-dimensional joint point map to get the two-dimensional joint point map of the occluded human body motion In the figure, the ordinate represents time, the abscissa represents joint points, and each point in the figure stores the coordinates of the corresponding two-dimensional joint point at the corresponding time. This representation method can quickly synthesize a large amount of occlusion data without affecting the generalization ability of the model on real data.

[0069] S14. Extract human 3D motion sequences from existing human data sets, where each frame of human 3D motion is represented by a skeleton-skinned 3D human model. For a 24-frame human 3D motion sequence with 24 joints, the skeleton posture of the 3D human model is represented by joint point rotation, and the human mesh is deformed by the joint point rotation and stored in the 3D complete human motion map I. 3d ∈R 24×24×6 In. Figure 7 As shown, the ordinate in the figure represents time, the abscissa represents the joint point, and each point in the figure stores the coordinates of the three-dimensional joint point corresponding to the corresponding time.

[0070] S21. Figure 1 As shown in the figure, since considering the temporal relationship and spatial relationship separately at different stages will lead to information loss, the constructed encoder contains two modules: a local joint-level spatiotemporal feature extraction network and a global motion-level spatiotemporal feature extraction network to extract spatiotemporal features at different levels. The local spatiotemporal relationship module is used to model local spatiotemporal features. It contains four convolutional layers, of which the first three convolutional layers are convolutional layers with expansion coefficients of 1, 2, and 5, respectively. The outputs of the three expansion convolutional layers are concatenated and then fused with spatiotemporal features through the last convolutional layer.

[0071] S22. Due to the limited performance of the convolutional layer in terms of temporal continuity, a global motion-level spatiotemporal feature extraction network is further constructed to model global spatiotemporal features. The global motion-level spatiotemporal feature extraction network includes a global spatial relationship module and a global temporal relationship module, and adopts a Transformer network structure. Spatial embedding is added to the output of the first module, and temporal embedding is added to the output of the second module.

[0072] S23. Construct an encoder consisting of a fully connected network to regress the joint point map of the two-dimensional complete human motion from the encoding estimated by the encoder.

[0073] S31. As data augmentation plays a very important role in network training, the existing data is augmented by data augmentation. The specific strategies include: 1) Mirror flip. As the human body has bilateral symmetry, the human body data is multiplied by exchanging the rotation angles of symmetrical joints; 2) Sampling at different rates. A new human body motion sequence is obtained by sampling the original human body motion sequence at different sampling rates; 3) Reverse sampling. The order of the original human body motion sequence is reversed as a new motion sequence.

[0074] S32. Take the synthesized two-dimensional occluded human motion joint point graph as input, and output the two-dimensional complete human motion joint point graph through the occluded human spatiotemporal prior network. The two-dimensional complete human motion joint point graph output by the occluded human spatiotemporal prior network is supervised by the true value of the two-dimensional complete human motion joint point graph. L1 loss is used as a constraint, and the formula is as follows:

[0075]

[0076] Among them I 2d and They represent the reconstructed joint point graph of the two-dimensional complete human motion and the true value of the joint point graph of the two-dimensional complete human motion, ‖·‖1 represents the one-norm, and L2d Represents the output 2D full human motion joint point graph and the L1 loss of the corresponding ground truth.

[0077] In addition, the smoothness term is further used to constrain the joint point graph of the output two-dimensional complete human motion:

[0078]

[0079] in and They represent the two-dimensional joint points estimated at the current moment and the next moment, that is, the parameters of the t-th row and t+1-th row of the joint point diagram of the two-dimensional complete human motion. smooth,2d Smoothness constraints on the joint graph representing the output 2D full human motion.

[0080] Finally, the training loss of the occluded human spatiotemporal prior network is:

[0081] L self =ω1L 2d +ω2L smooth,2d

[0082] Among them, ω1=1, ω2=0.5 are weights.

[0083] S41. Construct the same network as the spatiotemporal prior network encoder for occluded human body in S2 as the encoder, and use a fully connected layer and two Transformer modules for estimating temporal and spatial relationships respectively as the decoder to estimate the 3D complete human body motion map from the 2D occluded human body motion joint point map.

[0084] S51. The parameters of the spatiotemporal prior network encoder of the occluded human body trained in step S3 are used as pre-training parameters, and assigned to the encoder of the three-dimensional motion reconstruction network constructed in S4, so that the three-dimensional motion reconstruction network obtains the prior knowledge of the occluded two-dimensional human body motion.

[0085] S52. Figure 2 As shown in the figure, the synthesized 2D occluded human motion joint point map is used as input, and after being encoded and decoded by the 3D motion reconstruction network, a 3D complete human motion map is output. The output 3D complete human motion map is constrained using the reconstructed L2 loss, and the formula is as follows:

[0086]

[0087] Among them I 3d ,V 3d ,J 3d are the reconstructed complete 3D human motion graph, vertex coordinates and joint coordinates, β is the parameter representing the human body shape, the superscript ^ represents the true value of each parameter obtained from the data set, and L recrepresents the L2 loss of reconstruction.

[0088] Furthermore, an additional regularization constraint is used to prevent abnormal human body shapes:

[0089] L reg =‖β‖2

[0090] In addition, the output 3D complete human motion graph is further constrained by a smoothing term:

[0091] L smooth,3d =‖M(θ)-M(θ gt )‖2+‖M(θ)‖2

[0092] where θ is the value from I 3d The human motion parameters sampled in , M(θ) is the motion feature obtained by motion prior encoding, L smooth,3d Represents the smoothness term constraint on the output 3D complete human motion graph.

[0093] Finally, the training loss of the 3D motion reconstruction network is:

[0094] L 3dbranch =L rec +ω3L reg +ω4L smooth,3d

[0095] Among them, ω3=1, ω4=0.5 are weights. During the training process, the encoder parameters are fine-tuned at the same time. The training is continued until convergence.

[0096] S61. Sample a 3D human motion sequence from the 3D complete human motion map estimated in S4. Since the sampled 3D human motion sequence is in a local coordinate system, in order to obtain a 3D human motion sequence with an absolute position, the joint point positions of the 3D human sequence skeleton and the input occluded 2D joint point positions are used to construct a least squares function to estimate the global translation. The estimated translation is applied to the 3D human motion sequence. The following least squares function is constructed:

[0097]

[0098] Where K is the camera internal parameter, P is the coordinates of the two-dimensional joint points of the human body sampled and restored in the joint point map of the two-dimensional occluded human body motion, ω c is the confidence of the corresponding joint point, It is the translation amount of the skeleton-skinned 3D human body model. By adding the translation amount to the skeleton-skinned 3D human body model, the 3D motion of the human body with absolute position can be obtained.

[0099] S71. Use a single camera to capture video. Use a tripod to fix the camera and capture the obstructed human body.

[0100] S72. Figure 3 As shown, the video sequence collected in step S71 is input, and the joint point position and its confidence are detected for each frame of the picture using the existing open source two-dimensional joint point detection method. The joint point position is stored as a two-dimensional joint point map according to the method in S13. The confidence is binarized, and the threshold is set to 0.6. The value greater than the threshold is assigned to 1, and the value less than the threshold is assigned to 0, and the result is stored in a 24×24×1 confidence map. The confidence map and the two-dimensional joint point map are multiplied to obtain a two-dimensional joint point map that occludes human body motion.

[0101] S73. Input a two-dimensional occluded human body motion joint point diagram, and output a reconstructed three-dimensional complete human body motion diagram through the occluded human body spatiotemporal prior network and the three-dimensional motion reconstruction network.

[0102] S74. Obtain the global position of each posture through S61. Finally, Figure 8 As shown in the figure, by adding translation parameters to the skeleton-skinned 3D human body model, the 3D human body motion with absolute position is obtained. The global human body motion sequence can be further skinned to obtain a deformable human body network model.

Claims

1. A method for reconstructing human motion from a single perspective based on self-supervised spatiotemporal motion priors, characterized in that: The following steps are involved: S1. Human motion synthesis and representation: A two-dimensional occluded human motion joint point graph representation and a human occlusion data generation method are proposed to quickly synthesize a large amount of occlusion data; occluders are obtained by using instance segmentation in network videos, and the occluders are randomly covered on the pictures in the human data set without occlusion to achieve synthetic occlusion, and a two-dimensional occluded human motion joint point graph is generated on the two-dimensional complete human motion joint point graph for network training; S2. Construction of spatiotemporal prior network for occluded human body: Construct an encoder including a spatiotemporal feature extraction network at the local joint level and a spatiotemporal feature extraction network at the global motion level to extract spatiotemporal features at different levels respectively; in addition, construct an encoder composed of a fully connected network to regress the joint point map of the two-dimensional complete human body motion from the encoding estimated by the encoder; S3. Training of the spatiotemporal prior network for occluded human bodies: taking the synthesized two-dimensional occluded human body motion joint point graph as input and the corresponding two-dimensional complete human body motion joint point graph without occlusion as supervision, the spatiotemporal prior network for occluded human bodies constructed in step S2 is trained in a self-supervised manner; the neural network is trained in a self-supervised manner until convergence, and the prior knowledge of occluded two-dimensional human body motion is learned; S4. Construction of 3D motion reconstruction network: construct the same network as the spatiotemporal prior network encoder of the occluded human body in step S2 as the encoder, and use the fully connected layer and two Transformer modules as the decoder to estimate the 3D complete human body motion map from the joint point map of the 2D occluded human body motion; S5. 3D motion reconstruction network training: the parameters of the occluded human body spatiotemporal prior network encoder trained in step S3 are used as pre-training parameters, and assigned to the encoder of the 3D motion reconstruction network constructed in step S4, so that the 3D motion reconstruction network obtains the prior knowledge of the occluded 2D human body motion; further, the synthesized 2D occluded human body motion joint point map is used as input, and the 3D complete human body motion map is used as supervision to perform network training; During the training process, the encoder parameters are fine-tuned at the same time; training is performed until convergence; S6. Global position estimation: sampling a 3D human motion sequence from the 3D complete human motion map estimated in step S4; since the sampled 3D human motion sequence is in a local coordinate system, in order to obtain a 3D human motion sequence with an absolute position, a least squares function is constructed using the joint point positions of the 3D human sequence skeleton and the input occluded 2D joint point positions to estimate the global translation; and applying the estimated translation to the three-dimensional human motion sequence; S7. Real-time single-view occluded human motion reconstruction: After completing the network training, a single-view motion capture system is constructed using a single RGB camera. A human motion video sequence captured by a single-view camera is input. The existing open-source two-dimensional joint point detection method is used to detect the joint point position and its confidence for each frame of the image to obtain a two-dimensional joint point map of the occluded human motion; a three-dimensional complete human motion map is estimated through a three-dimensional motion reconstruction network, and a three-dimensional human motion sequence with absolute position is obtained using the method in step S6 to achieve single-view occluded human three-dimensional motion reconstruction. The global human motion sequence is further skinned to obtain a deformable human network model; The step S2 specifically includes: S21. Considering the temporal relationship and spatial relationship separately at different stages will lead to information loss. Therefore, the encoder constructed contains two modules: a local joint-level spatiotemporal feature extraction network and a global motion-level spatiotemporal feature extraction network to extract spatiotemporal features at different levels. The local spatiotemporal relationship module is used to model local spatiotemporal features. It contains four convolutional layers, of which the first three convolutional layers are convolutional layers with expansion coefficients of 1, 2, and 5, respectively. The outputs of the three expansion convolutional layers are concatenated and then fused with spatiotemporal features through the last convolutional layer. S22. Due to the limited performance of the convolutional layer in terms of temporal continuity, a global motion level spatiotemporal feature extraction network is further constructed to model global spatiotemporal features; the global motion level spatiotemporal feature extraction network includes a global spatial relationship module and a global temporal relationship module, and adopts a Transformer network structure; spatial embedding is added to the output of the first module, and temporal embedding is added to the output of the second module; S23. Construct an encoder consisting of a fully connected network to regress the joint point map of the two-dimensional complete human motion from the encoding estimated by the encoder.

2. The method for reconstructing human motion from a single perspective based on self-supervised spatiotemporal motion prior according to claim 1, characterized in that: The step S1 specifically includes: S11. Random occluder acquisition: Obtain occluder masks and RGB images using instance segmentation in online videos through Mask-RCNN to simulate occlusion; S12. Acquisition of joint point graph of 2D complete human motion: Extract the 2D human motion sequence from the human data set, where the 2D human motion of each frame is represented by the coordinates of K human joint points; for the 2D human motion sequence with K joint points in F frames, first subtract the root node coordinate (x) from all the related node coordinates (x, y). root ,y root ) is then divided by the skeleton bounding box size w×h to achieve normalization, and the normalized point coordinates are Stored in a two-dimensional joint point graph I 2d ∈R F×K×2 In the figure, each row represents the normalized position of all nodes in a frame. S13. Acquisition of joint point graph of two-dimensional occluded human motion: Obtain occluded human motion data by adding occluders to the non-occluded human image; Calculate the intersection-and-union ratio between the skeleton bounding box and the occluders obtained in S11, and make the intersection-and-union ratio between X1 and X2 to achieve occlusion of different occlusion ratios; For the mask image of the occluder, by stipulating that the value of the joint points located inside the mask image is 0, and the value of the joint points located outside the mask image is 1, a confidence map I will be generated c ∈R F×K×1 ; Multiply the confidence map and the two-dimensional joint point map to obtain the two-dimensional joint point map of the occluded human body motion S14. Acquisition of a 3D complete human motion map: extract a 3D human motion sequence from an existing human data set, where the 3D human motion of each frame is represented by a 3D human model with a skeleton skin; for a 3D human motion sequence with F frames and N joints, the skeleton posture of the 3D human model is represented by the rotation of the joints, and the human mesh is deformed by the rotation of the joints, which is stored in the 3D complete human motion map I. 3d ∈R F×N×6 middle.

3. The method for reconstructing human motion from a single perspective based on self-supervised spatiotemporal motion prior according to claim 1, characterized in that: The step S3 specifically includes: S31. In order to enhance the generalization performance of the network, data augmentation is performed on the training data. The specific strategies include: 1) Mirror flipping: Since the human body has bilateral symmetry, the human body data is multiplied by exchanging the rotation angles of symmetrical joints; 2) Sampling at different rates: A new human body motion sequence is obtained by sampling the original human body motion sequence at different sampling rates; 3) Reverse sampling: The order of the original human body motion sequence is reversed as a new motion sequence; S32. The synthesized two-dimensional occluded human motion joint point map is used as input, and the two-dimensional complete human motion joint point map is output through the occluded human spatiotemporal prior network; the two-dimensional complete human motion joint point map output by the occluded human spatiotemporal prior network is supervised by the true value of the two-dimensional complete human motion joint point map; L1 loss is used as a constraint, and the formula is as follows: Among them I 2d and They represent the reconstructed joint point graph of the two-dimensional complete human motion and the true value of the joint point graph of the two-dimensional complete human motion, ‖·‖1 represents the one-norm, and L 2d Represents the output two-dimensional complete human motion joint point map and the L1 loss of the corresponding true value; In addition, the smoothness term is further used to constrain the joint point graph of the output two-dimensional complete human motion: in and They represent the two-dimensional joint points estimated at the current moment and the next moment, that is, the parameters of the t-th row and t+1-th row of the joint point diagram of the two-dimensional complete human body motion; L smooth,2d A smoothness constraint on the joint point graph representing the output two-dimensional complete human motion; Finally, the training loss of the occluded human spatiotemporal prior network is: L self =ω1L 2d +ω2L smooth,2d Where ω1 and ω2 are weights.

4. The method for reconstructing human motion from a single perspective based on self-supervised spatiotemporal motion prior according to claim 1, characterized in that: The step S4 specifically includes: S41. Construct a network identical to the spatiotemporal prior network encoder for the occluded human body in step S2 as an encoder, and use a fully connected layer and two Transformer modules for estimating temporal and spatial relationships, respectively, as a decoder to estimate a three-dimensional complete human body motion map from a two-dimensional occluded human body motion joint point map.

5. The method for reconstructing human motion from a single perspective based on self-supervised spatiotemporal motion prior according to claim 1, characterized in that: The step S5 specifically includes: S51. The parameters of the occluded human body spatiotemporal prior network encoder trained in step S3 are used as pre-training parameters and assigned to the encoder of the three-dimensional motion reconstruction network constructed in step S4, so that the three-dimensional motion reconstruction network obtains the prior knowledge of the occluded two-dimensional human body motion; S52. The synthesized two-dimensional occluded human body motion joint point map is used as input, and after being encoded and decoded by the three-dimensional motion reconstruction network, a three-dimensional complete human body motion map is output; the output three-dimensional complete human body motion map is constrained using the reconstructed L2 loss, and the formula is as follows: Among them I 3d ,V 3d ,J 3d are the reconstructed complete 3D human motion graph, vertex coordinates and joint coordinates, β is the parameter representing the human body shape, the superscript ^ represents the true value of each parameter obtained from the data set, and L rec represents the L2 loss of reconstruction; Furthermore, an additional regularization constraint is used to prevent abnormal human body shapes: L reg =‖β‖2 In addition, the output 3D complete human motion graph is further constrained by a smoothness term: where θ is the value from I 3d The human motion parameters sampled in , M(θ) is the motion feature obtained by motion prior encoding, L smooth,3d Represents the smoothness constraint on the output three-dimensional complete human motion map; Finally, the training loss of the 3D motion reconstruction network is: L 3dbranch =L rec +ω3L reg +ω4L smooth,3d Among them, ω3 and ω4 are weights. During the training process, the encoder parameters are fine-tuned at the same time, and the training is carried out until convergence.

6. The method for reconstructing human motion from a single perspective based on self-supervised spatiotemporal motion prior according to claim 1, characterized in that: The step S6 specifically includes: S61. Sampling a 3D human motion sequence from the 3D complete human motion map estimated in step S4; since the sampled 3D human motion sequence is in a local coordinate system, in order to obtain a 3D human motion sequence with an absolute position, a least squares function is constructed using the joint point positions of the 3D human sequence skeleton and the input occluded 2D joint point positions to estimate the global translation; and the estimated translation is applied to the 3D human motion sequence; the following least squares function is constructed: Where K is the camera internal parameter, P is the coordinates of the two-dimensional joint points of the human body sampled and restored in the joint point map of the two-dimensional occluded human body motion, ω c is the confidence of the corresponding joint point, It is the translation amount of the skeleton-skinned three-dimensional human body model; by adding the translation amount to the skeleton-skinned three-dimensional human body model, the three-dimensional movement of the human body with absolute position is obtained.

7. The method for reconstructing human motion from a single perspective based on self-supervised spatiotemporal motion prior according to claim 6, characterized in that: The step S7 specifically includes: S71. Use a single camera to collect video; fix the camera with a tripod to collect the obstructed human body; S72. Use the existing open-source two-dimensional joint point detection method to detect the joint point position and its confidence for each frame of the video sequence collected in step S71; store the joint point position as a two-dimensional joint point map according to the method in S13; binarize the confidence, assign 1 to the value greater than the threshold value X, and assign 0 to the value less than the threshold value X, and store the result in a confidence map of F×N×1; multiply the confidence map and the two-dimensional joint point map to obtain a two-dimensional joint point map of occluded human body motion; S73. Input a two-dimensional occluded human body motion joint point diagram, and output a reconstructed three-dimensional complete human body motion diagram through the occluded human body spatiotemporal prior network and the three-dimensional motion reconstruction network; S74. Obtain the global position of each posture through step S61; finally, obtain the three-dimensional motion of the human body with absolute position by adding translation parameters to the three-dimensional human body model with skeleton skin; further skin the global human body motion sequence to obtain a deformable human body network model.