3D Human Reconstruction Method Based on Temporal Context Clues

Abstract

The present invention discloses a method for 3D human body reconstruction based on temporal context clues, specifically relating to the field of artificial intelligence. The method mainly includes the following steps: pre-training a neural network for 3D human body reconstruction based on temporal context clues; extracting spatial features from each frame of an image using a convolutional neural network; adding the external contour and optical flow information of the human body in each frame to the input features using a motion encoder; using a transformer network as a temporal encoder to capture the temporal correlation of multi-frame inputs; regressing parameterized human body template parameters and camera parameters through training a regressor; using a discriminator to discriminate realistic and natural human motion postures; and training a differentiable renderer using the parameterized human body template parameters obtained through regression. After training, the posture and shape of a human body model can be reconstructed given any sequence of motion. This technology can be used in scenarios requiring 3D human body reconstruction, such as motion analysis, virtual and augmented reality, games, and animation.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and more particularly, to a three-dimensional human body reconstruction method based on temporal context clues. Background Art

[0002] Reconstructing the pose and shape of a human body model from monocular video is an important problem in computer vision and artificial intelligence. Generating accurate and smooth 3D human body reconstruction results has broad application prospects and important application value in the fields of virtual and augmented reality.

[0003] In recent years, with the integration and development of deep learning technology and computer vision technology, 3D reconstruction methods based on deep neural networks have emerged. However, due to the lack of datasets in natural scenes with 3D human body annotations, existing temporal models of human motion do not capture the complexity and variability of real human motion; secondly, since the average error of each joint position only penalizes spatial errors without considering temporal consistency, the pose estimation exhibits "jittering" phenomena, making it difficult for the results to approach the true pose; indoor 3D human datasets are limited in terms of the number of targets, motion range, and image complexity. Summary of the Invention

[0004] In order to overcome the above-mentioned defects of the prior art, the present invention provides a three-dimensional human body reconstruction method based on temporal context clues. By introducing the deep neural network three-dimensional human body reconstruction method into monocular video, on the one hand, a convolutional neural network is used to extract spatial information from the video sequence, and on the other hand, a converter network is used to capture the temporal correlation of multi-frame inputs, and finally a feature containing the spatiotemporal information of the entire input is obtained. At the same time, the quality and accuracy of the reconstruction of the human body model posture and shape are further improved by combining temporal context clues such as optical flow and contour, so as to achieve smooth and natural and realistic human body model posture and shape reconstruction to solve the problems raised in the above-mentioned background technology.

[0005] To achieve the above object, the present invention provides the following technical solutions:

[0006] The 3D human body reconstruction method based on temporal context clues includes the following steps:

[0007] Step S1: Input a single human body video frame sequence, denoted as ,in is the sequence length, i.e. the number of image sequences, Indicates the Image, i.e. frame i;

[0008] Step S2: Use convolutional neural network to process each frame of the image sequence. Extracting spatial features , where each feature is a vector of size 2048;

[0009] Step S3: Add the human body external contour and optical flow information of each frame to the input feature through a motion encoder to obtain the feature value: ,in is the spatial feature of the video frame, The external contour features of the human body, is the optical flow feature;

[0010] Step S4: Train a transformer network as a temporal encoder to extract contextual time information and output a latent variable containing the previous and next frame information for each frame. ;

[0011] Step S5: Utilize Regression parameterized human template parameters and camera parameters and , the regressor is initialized to the average pose , followed by the posture result of the previous frame As the next frame initialization, the corresponding 3D human body template dynamic sequence is fitted to the entire sequence ;

[0012] Step S6: using self-supervision to integrate the feature vectors corresponding to each time to discriminate the real and natural human body motion posture;

[0013] Step S7: Perform differentiable rendering using the parameterized human body template parameters obtained through regression, compare the obtained two-dimensional joint point information, human body contour, and optical flow information with the network-estimated values, and calculate the reprojection error;

[0014] Step S8: Construct a loss function using the human body template posture sequence and all image and video frame sequences , train the network model;

[0015] Step S9: After the training in step S8 is completed, given any video frame sequence, the posture and shape of the three-dimensional human body model are reconstructed using the trained model.

[0016] In a preferred embodiment, in step S5, the fitted three-dimensional human body template is a linear function , linear function The input is the posture parameters of the human body , the output is the vertex coordinates of the 3D human body template ,Right now ,in is the total number of vertices of the 3D human body template; from the output 3D human body template vertex coordinates, the joint point coordinates of the human body template can be regressed: ,in is the regression matrix.

[0017] In a preferred embodiment, in step S8, the loss function for:

[0018]

[0019] Among them, the three-dimensional error , using the L2 norm loss function:

[0020] ,in, is the 3D joint parameter

[0021] Two-dimensional error , using the L2 norm loss function:

[0022] ,in, is the two-dimensional joint parameter

[0023] Parameterized human body template error , using the L2 norm loss function:

[0024] ,in, are the body shape parameters, is the human body posture parameter

[0025] Discriminator error , using the L2 norm loss function:

[0026] ,in, is the adversarial loss of motion parameters

[0027] Discriminator error , using the L2 norm loss function:

[0028]

[0029] Motion encoder error , using the L2 norm loss function:

[0030] ,in, The external contour features of the human body, is the optical flow feature, is the spatial feature of the video frame.

[0031] The technical effects and advantages of the present invention are as follows:

[0032] The present invention is based on a 3D reconstruction method based on temporal context clues, which introduces a deep neural network 3D reconstruction method into monocular videos. On the one hand, a convolutional neural network is used to extract spatial information from video sequences, and on the other hand, a transformer network is used to capture the temporal correlation of multi-frame inputs, ultimately obtaining features containing the spatiotemporal information of the entire input. At the same time, the quality and accuracy of human body model posture and shape reconstruction are further improved by combining temporal context clues such as optical flow and contours, so as to achieve smooth, natural and realistic human body model posture and shape reconstruction. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] Figure 1 This is a network structure diagram of the three-dimensional human body reconstruction method based on temporal context clues of the present invention.

[0034] Figure 2 This is a flow chart of the three-dimensional human body reconstruction method based on temporal context clues of the present invention. DETAILED DESCRIPTION

[0035] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0036] This paper presents a 3D human body reconstruction method based on temporal context clues, focusing on introducing deep neural network 3D reconstruction methods into monocular videos. On the one hand, it uses convolutional neural networks to extract spatial information from video sequences, and on the other hand, it uses transformer networks to capture the temporal correlation of multi-frame inputs, ultimately obtaining features containing the spatiotemporal information of the entire input. At the same time, it combines temporal context clues such as optical flow and contours to further improve the quality and accuracy of human body model posture and shape reconstruction, achieving smooth, natural and realistic human body model posture and shape reconstruction.

[0037] To this end, this paper utilizes a spatiotemporal encoder and a motion encoder to extract human motion features, capturing the temporal correlations of multi-frame information. Unlike existing methods, this method uses a transformer network to capture the inherent correlations of long temporal inputs during spatial feature extraction, ultimately generating features that encompass the spatiotemporal information of the entire input. This method also integrates motion and contour information to predict human body model parameters.

[0038] Specifically, when the present invention is applied to reconstructing the posture and shape of a human body model in a video, the present invention includes the following steps:

[0039] like Figure 1As shown, self-supervision serves as the core of the transformer network, linking the motion features of the input image frame sequence with the embedded features. Our transformer network consists of multi-way self-supervision and multi-layer perceptrons. A normalization layer is applied before each module, and a residual connection is applied after each module.

[0040] The attention module can be described as a mapping function that transforms the query matrix , key matrix Sum Matrix Mapped to the output attention matrix. ,in is the number of vectors in the sequence, is the dimension. The output of the attention module can be expressed as:

[0041]

[0042] In this article , . and is the embedded feature Through linear transformation and Calculated:

[0043]

[0044] Multi-way self-supervision uses multiple branches to model information in the representation subspace at different locations. Each branch applies the attention module in parallel. The MSA output will be Connect the outputs of multiple self-supervised channels:

[0045]

[0046]

[0047] Input embedding features ,have The transformer network structure of the layer can be expressed as:

[0048]

[0049]

[0050]

[0051] in is the layer-level normalization operation. The output of the converter network Keep the same size as the input. For prediction, the encoder output Compress to vector , and averaged in the frame dimension. Finally, the output is regressed to through a multi-layer perceptron layer.

[0052] The 3D human body template used in this article is a linear function The input of this function is the posture parameters of the human body, that is, the rotation of the skeleton joints, and the output is the vertex coordinates of the three-dimensional human body template. ,Right now , where is the total number of vertices of the 3D human body template. From the output 3D human body template vertex coordinates, the joint coordinates of the human body template can be regressed: ,in is the regression matrix.

[0053] in = (here is true value), is the current time The posture parameters (i.e., the rotation of the human joints) are the predicted parameters of a single person's body shape (i.e., height, weight, etc.). The parameters are obtained by averaging the shape parameters of each frame.

[0054] The loss function of the entire model for:

[0055]

[0056] Among them, the three-dimensional error , using the L2 norm loss function:

[0057] ,in, is the 3D joint parameter

[0058] Two-dimensional error , using the L2 norm loss function:

[0059] ,in, is the two-dimensional joint parameter

[0060] Parameterized human body template error , using the L2 norm loss function:

[0061] ,in, are the body shape parameters, is the human body posture parameter

[0062] Discriminator error , using the L2 norm loss function:

[0063] ,in, is the adversarial loss of motion parameters

[0064] Discriminator error , using the L2 norm loss function:

[0065]

[0066] Motion encoder error , using the L2 norm loss function:

[0067] ,in, The external contour features of the human body, is the optical flow feature, is the spatial feature of the video frame.

[0068] Specifically, such as Figure 2 As shown in FIG, the specific steps of the 3D human body reconstruction method based on temporal context clues are as follows:

[0069] Step S101, pre-training of a neural network for 3D human body reconstruction based on temporal context clues, mainly including a spatial encoder, a temporal encoder, a motion encoder, a regressor, and a differentiable renderer. The dataset includes a mixed 2D and 3D dataset, 5,000 video datasets with 2D true values, and 8,000 pseudo-label datasets obtained using a 2D key point detector. For the 3D dataset, 2,000 video data with true values of parameterized human body templates are used for calculation.

[0070] In step S102, a convolutional neural network is used to extract spatial features from each frame of the image sequence, where each feature is a vector of 2048 dimensions. The specific network is a 50-layer residual network, and the final output feature size is 2048 dimensions, the sequence length, and the batch size is 32.

[0071] In step S103, the human body external contour and optical flow information of each frame are added to the input features through a motion encoder to obtain the feature value of the point: ,in is the spatial feature of the video frame, The external contour features of the human body, is the optical flow feature.

[0072] In step S104, the updated features are then passed to the converter network encoding layer. The network architecture of the model consists of two parts: a self-attention mechanism and a shallow fully connected feedforward network. The output of each part is processed by residual network and hierarchical normalization. The temporal encoder composed of the converter network extracts contextual time information and outputs latent variables containing the previous and next frame information for each frame.

[0073] In step S105, the human body model parameters and camera parameters are parameterized by regression. The regressor is initialized to the average posture. The posture result of the previous frame is then used as the initialization for the next frame. The corresponding three-dimensional human body template dynamic sequence is fitted to the entire sequence. The parameterized human body template regressor consists of two fully connected layers, each with 1024 neurons. Finally, the final layer with a dimension of 85 is output, which contains information such as posture, shape and camera parameters.

[0074] In step S106, multi-channel self-supervision is used to integrate the feature vectors corresponding to each time to discriminate real and natural human motion postures. Two multi-layer perceptron layers are used, each with 1024 neurons and sinusoidal activation to learn attention weights. Finally, the linear layer predicts whether each sample belongs to a real and reasonable human motion posture.

[0075] Step S107 , performing differentiable rendering using the parameterized human body model parameters obtained by regression, comparing the obtained two-dimensional joint point information, human body contour, and optical flow information with the network estimated values, and calculating the reprojection error.

[0076] Step S108: construct a loss function using the human body template posture sequence and all images to train the network model.

[0077] Step S109: During the training process, an adaptive moment estimation optimizer is used, the learning rate is fixed at 0.0001, and 120 rounds of training are performed. The evaluation indicators include the average per-joint position error, the average per-joint position error, the percentage of correct key points, the vertex-by-vertex error, and the acceleration error. The acceleration error is calculated based on the difference between the true value and the predicted acceleration of each joint's three-dimensional coordinate point. The unit is , which is the main smoothness indicator for estimating motion sequences. A better acceleration error indicates a smoother and more natural human motion estimation.

[0078] Step S110: After the training is completed, given any video frame sequence, the posture and shape of the three-dimensional human body model can be reconstructed using the trained model.

[0079] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A 3D human body reconstruction method based on temporal context clues, characterized by: The steps include: Step S1: Input a single human body video frame sequence, denoted as ,in is the sequence length, i.e. the number of image sequences, Indicates the Image, i.e. frame i; Step S2: Use convolutional neural network to process each frame of the image sequence. Extracting spatial features , where each feature is a vector of size 2048; Step S3: Add the human body external contour and optical flow information of each frame to the input feature through a motion encoder to obtain the feature value: ,in is the spatial feature of the video frame, The external contour features of the human body, is the optical flow feature; Step S4: Train a transformer network as a temporal encoder to extract contextual time information and output a latent variable containing the previous and next frame information for each frame. ; Step S5: Utilize Regression parameterized human template parameters and camera parameters and , the regressor is initialized to the average pose , followed by the posture result of the previous frame As the next frame initialization, the corresponding 3D human body template dynamic sequence is fitted to the entire sequence ; Step S6: using self-supervision to integrate the feature vectors corresponding to each time to discriminate the real and natural human body motion posture; Step S7: Perform differentiable rendering using the parameterized human body template parameters obtained through regression, compare the obtained two-dimensional joint point information, human body contour, and optical flow information with the network-estimated values, and calculate the reprojection error; Step S8: Construct a loss function using a 3D human body template dynamic sequence and a single human body video frame sequence ; Training network model, loss function for: ； Among them, the three-dimensional error , using the L2 norm loss function: ,in, are the three-dimensional joint parameters; Two-dimensional error , using the L2 norm loss function: ,in, is the two-dimensional joint parameter; Parameterized human body template error , using the L2 norm loss function: ,in, are the body shape parameters, is the human body posture parameter; Discriminator error , using the L2 norm loss function: ,in, is the adversarial loss of motion parameters; Discriminator error , using the L2 norm loss function: ； Motion encoder error , using the L2 norm loss function: ,in, The external contour features of the human body, is the optical flow feature, is the spatial feature of the video frame; Step S9: After the training in step S8 is completed, given any video frame sequence, the posture and shape of the three-dimensional human body model are reconstructed using the trained model.

2. The method for 3D human body reconstruction based on temporal context clues according to claim 1, characterized in that: In step S5, the fitted 3D human body template is a linear function The input of this function is the posture parameter of the human body, that is, the rotation amount of the skeleton joint , the output is the vertex coordinates of the 3D human body template ,Right now ,in is the total number of vertices of the 3D human body template; from the output 3D human body template vertex coordinates, the joint point coordinates of the human body template can be regressed: ,in is the regression matrix.

Images