A method for reconstructing human body model based on encoder-decoder network

Abstract

The present invention discloses a method for reconstructing a human body model based on a codec network. The present invention uses the point cloud data collected by a depth camera as input, corrects the model generated by the input data to a T-shaped standard posture through a codec network, and predicts the model shape vector and posture vector parameters through a constructed neural network model, and then applies the predicted parameters to a human body parametric model, thereby reconstructing a clear and complete human body model. The present invention has a short processing time and a simple algorithm, greatly improves the calculation efficiency of the model, improves the accuracy of the T-shaped standard posture, and the collected object does not need to maintain the standard posture during the collection process, and the collection difficulty is greatly reduced.

Description

Technical Field

[0001] The invention belongs to the technical field of human body measurement in clothing customization industry, and in particular relates to a method for reconstructing a human body model using a codec network. Background Art

[0002] With the advent of the Internet+ era, industries such as computers, clothing manufacturing, and film and television animation are rapidly developing towards intelligence. 3D reconstruction technology has led a new direction for the clothing industry. At present, the acquisition of human body size data generally adopts the traditional point cloud modeling method, and the original data captured by the camera is post-processed to obtain 3D human body model data. However, the existing post-processing operations such as denoising, hole filing, and dislocation have problems such as complex processing and inaccurate results.

[0003] Since the 21st century, deep learning technology has developed rapidly, and deep learning technology has also provided sufficient impetus for the transformation and upgrading of the clothing customization industry. For example, deep learning can conduct autonomous training to adjust network parameters after network initialization. This new post-processing method not only saves manpower but also has high computational efficiency. However, the existing training models used in clothing customization often have very complex algorithms, are time-consuming, and require additional posture correction, etc., and there is still a lot of room for optimization. Summary of the invention

[0004] Aiming at the common problems of existing human body three-dimensional measurement post-processing technology, such as large amount of calculation, long time consumption, and inaccurate post-processing results, the present invention proposes a method for reconstructing a human body model based on a codec network.

[0005] The present invention takes the point cloud data collected by the depth camera as input, corrects the model generated by the input data to the T-shaped standard posture through the encoder-decoder network, and predicts the model shape vector and posture vector parameters through the constructed neural network model, and then applies the predicted parameters to the human body parametric model (SMPL) to reconstruct a clear and complete human body model.

[0006] To achieve the above object, the present invention comprises the following steps:

[0007] S1. A set of human point cloud data was generated by using the random combination of SMPL models and simulated depth cameras. The SMPL parameters were randomly selected to obtain human point cloud data sets of different human postures. The human point cloud data collected by the simulated camera was imported into the simulation software. The software parameters were adjusted to simulate the abnormal situations that may occur during the human data collection process. Finally, a set of data sets containing different postures of human models and various abnormal situations such as outliers, holes, point cloud dislocations, and posture problems were generated.

[0008] S2. Design the encoder-decoder network

[0009] S2.1. The encoder network records the human body model in the form of points. Each point contains the three-dimensional vector coordinates of the human body. The vector set of these three-dimensional vector coordinates is input as point cloud data. It first performs continuous convolution from a small range to capture local features, and then the local features are aggregated and converted into global feature vectors.

[0010] S2.2. The decoder network first estimates and predicts a rough T-shaped standard posture based on the global feature vector output by the encoder network, and then takes the rough T-shaped posture as input. The trained decoder estimates the shape vector and posture vector parameters of the human body model. The data contained in the training model is synthesized from randomly combined human body data sets. The standard human body posture can be obtained from the shape vector and posture vector.

[0011] S3. Input the predicted vector parameters into SMPL to obtain the reconstructed human body model.

[0012] Furthermore, the human body model reconstruction processing method based on the encoder-decoder network takes three-dimensional unstructured points as input, which is an improved variant of the point network. The encoder network consists of two stacked neural network architecture (PN) layers. The first PN layer processes the input point cloud data through two perceptrons and obtains point-by-point features. The perceptron is composed of neurons using the ReLU activation function. The obtained point-by-point features are concatenated with the input point cloud to form a set of feature point clouds. The second PN layer uses the point-by-point maximum pooling algorithm to convert the feature point cloud into a global feature vector.

[0013] Furthermore, the decoder network adopts a decoder c2f-D composed of a coarse-to-fine strategy. The c2f-D decoder includes a global shape reconstruction (GSR) module and a surface reconstruction (SR) module. The role of the decoder is to use the global feature vector obtained in the encoder network as input, predict the rough point cloud data, and give a rough T-pose. The SR module is a sub-encoder-decoder network used to achieve the reconstruction of the rough T-pose to the fine standard T-pose. Therefore, the SR encoder-decoder architecture is the same as the above encoder network structure, that is, the rough T-pose data is combined with the global feature vector as a new feature vector to predict the shape vector and posture vector parameters of the reconstructed SMPL.

[0014] Furthermore, the decoder network is trained by randomly combining the human body point cloud data and the human body point cloud data in T-pose. The human body pose is determined by the shape vector and attitude vector parameters of SMPL. Changing the parameters can change the human body pose. The SMPL parameters are randomly combined to simulate various human body poses, and the human body mesh in T-pose is used as the basic true value for training, forming training data for converting any human body pose into T-pose.

[0015] Furthermore, in order to simulate the outliers such as holes, low resolution, and outliers that appear in the depth camera, the human point cloud data can be imported into the point cloud simulation software Blensor. Adjusting the noise-sigma parameters in the software can generate a series of damaged data, and finally form the real outlier data of the depth camera.

[0016] Compared with the prior art methods, the technical advantages of the present invention are:

[0017] 1. Compared with other correction algorithms, the present invention predicts the data of T-shaped standard posture from the original human body data, eliminating the tedious processing process in the post-processing operation, and adopts the method of using the algorithm to predict the parameters of the shape vector and the posture vector to reconstruct a clear and complete human body model. The post-processing time of the present invention is short and the algorithm is simple.

[0018] 2. The present invention generates a series of outlier data by setting the noise-sigma parameters of the point cloud simulation software Blensor. The collection of data forms a new training data set. The trained model covers most of the data anomaly problems, greatly improves the calculation efficiency of the model, and improves the accuracy of the T-type standard posture.

[0019] 3. Compared with other human body data collection processes, the present invention proposes a new post-processing algorithm based on the encoder-decoder network to process any posture into a T-shaped standard posture. During the collection process, the collected object does not need to maintain the standard posture all the time, and the collection difficulty is greatly reduced. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] Figure 1 The overall architecture diagram of the encoder-decoder network;

[0021] Figure 2 Diagram of the process generated for training the dataset required. DETAILED DESCRIPTION

[0022] The present invention is further described below in conjunction with the accompanying drawings.

[0023] In order to obtain a clear and complete human body model, the present invention designs a codec network, the encoder converts the input human body point cloud data into a global feature vector, the decoder first corrects the global feature vector output by the encoder into a rough T-shaped posture, and then uses the rough T-shaped posture as input to predict the shape vector and posture vector parameters of the human body model. At the same time, in order to simulate various abnormal situations that occur during the human body data collection process, a multi-dimensional camera human body scanner is used to collect human body data, and the data is imported into the Blensor simulation software. The abnormalities of the human body data are simulated by changing the noise-sigma parameter, and various abnormal situations are integrated together to form a data set for training deep neural networks. Finally, a standard human body posture is given according to the real abnormal situation of the human body data, and a clear and complete human body model is finally reconstructed by SMPL.

[0024] The existing 3D scanning acquisition technology not only requires a standard human input posture, but also requires a series of necessary post-processing algorithm operations such as denoising, hole filing, surface reconstruction and surface smoothing, and is often accompanied by the problem of dislocation damage. The present invention uses a point cloud prediction algorithm to reconstruct the human body model and uses a multi-dimensional camera scanning method to shoot the human body in multiple directions. Compared with two-dimensional images, it avoids problems such as dislocation damage and greatly reduces the calculation amount and time consumption of the original method.

[0025] Example:

[0026] Take any set of human body point cloud data and use a shared multi-layer perceptron (MLP) to extract the features of each point to form the original body point set X K ={x k,i ∈R 3 ,i=1,…,N},x k,i It is the original human point cloud data.

[0027] In some embodiments, the encoder-decoder network structure diagram is as follows: Figure 1 As shown in the figure, the encoder model consists of two PN layers. The first PN layer uses an MLP composed of two linear layers with ReLU activation functions. All negative values ​​in the original data point coordinate X are changed to 0, and the positive values ​​remain unchanged to obtain a point-by-point feature matrix composed of a single point cloud feature. The point-by-point features are connected to the coordinates of the input point cloud to construct a new feature point cloud set.

[0028] The second PN layer takes the new feature point cloud processed by the first PN layer as input and outputs a new point-by-point feature matrix F (X，F) , apply the point-by-point maximum pooling algorithm on the point-by-point feature matrix to obtain the global point feature vector

[0029] Preferably, the first PN layer of the encoder is composed of a shared multilayer perceptron (MLP), and the MLP is composed of a large number of neurons to complete the reception, transmission and processing of information. The two-layer artificial neural network has 128 and 256 neurons respectively, and the second PN layer is composed of another shared two-layer MLP, and the two layers have 512 and 1024 neurons respectively.

[0030] The decoder network consists of c2f-D, which is responsible for generating the T-type standard pose from the feature vector f. c2f-D consists of two modules, GSR and SR.

[0031] Preferably, the c2f-D decoder follows a multi-stage reconstruction strategy to reconstruct a clear and complete human body mesh according to a coarse-to-fine framework, and reconstructs a clear and complete 3D human body model from a single image without the need to capture large-scale high-resolution data. In one embodiment, the following operations are performed:

[0032] In the first stage, the 3D rough point cloud Y is predicted through the encoder-decoder network based on the features of the encoder point cloud data. coarse , and obtain a rough T-pose; in the second stage, the decoder is used to extract the τ-parameterized point cloud feature E of the rough T-pose again τ ; In order to eliminate the rough reconstruction loss, the new feature E τ Associated with the encoder features, c2f-D predicts the SMPL vertices with the associated features, and finally obtains the high-fidelity T-shaped posture network Y fine , construct a reconstruction function with the obtained standard human body model and the rough human body model, and adjust the ground truth value Y in the training model GT , while reducing the loss of fine reconstruction, it improves the fine reconstruction effect of normal images, reduces false detection of normal images, and improves detection performance.

[0033] Figure 2 It is a process diagram for generating the data set required for training. In order to train the deep neural network, this embodiment uses a parameterized human body model (SMPL). The SMPL is determined by a shape vector β and a posture vector θ. Given a pair of specified β and θ, a new set body shape with 6890 vertices and 13776 triangular surfaces will be obtained.

[0034] Firstly, the β and θ parameters of the SMPL model are sampled from a large dataset SURREAL according to the distorted data. SURREAL contains 6 million synthetic human images created using the SMPL body model. Randomly combining SMPL parameters will cause corresponding changes in human posture. The randomly combined parameter values ​​are sampled by SMPL to obtain the true value of the human body to simulate data of various human postures.

[0035] Furthermore, by using the human body mesh of T posture as the ground truth value for training, the θ of its T posture is set as the zero vector. Next, a multi-dimensional human body scanner is placed in the front, back, left, and right directions of the human body, and the computer is used to simulate the scanning process to obtain the correct point cloud data X of the human body, where X is the complete human body point cloud data, T i , X i are the external matrix and partial point cloud data obtained by the i-th simulated scanner, respectively, and m is the number of scanners. The human body model of the scan result is imported into the Blensor simulation software, and the misalignment simulation of abnormal values ​​in the scan is performed by adjusting the parameter noise-sigma of the scan result to obtain the final misalignment data as the input for training the deep neural network.

[0036] This embodiment can also randomly select 99% and 0.7% of the synthetic data set for training and verification of the network, respectively, and the rest is used as a test data set to evaluate the performance of the algorithm. The Adam optimizer is used for training, with an initial learning rate of 0.0001, 50 cycles, and a batch size of 16. In order to verify the effectiveness of the deep model on real scan data, a data set consisting of 250 real human scan data is selected for verification. Abnormalities such as misalignment and holes of the human body model can be clearly observed in the data set. This model is applied to successfully predict a high-fidelity T-pose body model from these damaged scan data.

[0037] In the present invention, a human body measurement method based on a codec network is proposed. According to the raw data scanned by a depth camera as input, a high-fidelity human body model is reconstructed and the model posture of the post-processing result is standardized.

Claims

1. A method for reconstructing a human body model based on a codec network, characterized in that: The following steps are included: S1. Generate a set of human point cloud data by using the random combination of human body parameterized model SMPL and simulated depth camera, and randomly select SMPL parameters to obtain human point cloud data of different human body postures; Import the human body point cloud data into the simulation software, adjust the software parameters to simulate abnormal situations in the human body data collection process, and generate a set of human body point cloud data sets containing different postures of the human body model as well as outliers, holes, point cloud misalignment and posture problems; S2. Design the encoder-decoder network; S2.

1. The encoder network records the human body model in the form of points. Each point contains the three-dimensional vector coordinates of the human body. The vector set of the three-dimensional vector coordinates, i.e. the input human body point cloud data, is first convolved to capture local features. The local features are then aggregated and converted into global feature vectors. S2.2, the decoder network first predicts the rough T-pose from the global feature vector, and then uses the rough T-pose as input to estimate the shape vector parameters and posture vector parameters of the human body model by the trained decoder; S3. Input the shape vector parameters and posture vector parameters into SMPL to obtain the reconstructed human body model.

2. A method for reconstructing a human body model based on a codec network according to claim 1, characterized in that: The encoder network described in S2.1 is composed of two stacked neural network architecture PN layers. The first PN layer processes the input human body point cloud data through two perceptrons and obtains point-by-point features; The perceptron is composed of neurons using the ReLU activation function, and the obtained point-by-point features are concatenated with the input point cloud to form a set of feature point clouds; The second PN layer uses a point-by-point maximum pooling algorithm to convert the feature point cloud into a global feature vector.

3. A method for reconstructing a human body model based on a codec network according to claim 2, characterized in that: The decoder network described in S2.2 adopts a decoder c2f-D composed of a coarse-to-fine strategy. The c2f-D decoder includes a global shape reconstruction GSR module and a surface reconstruction SR module; The SR module is a sub-encoder-decoder network used to realize the reconstruction of the rough T-pose to the fine standard T-pose. The architecture of SR is the same as the encoder network structure mentioned above, that is, the rough T-pose data is combined with the global feature vector as a new feature vector to predict the shape vector parameters and posture vector parameters of the reconstructed SMPL.

4. The method for reconstructing a human body model based on a codec network according to claim 3, characterized in that: S2.2 also includes randomly combined human point cloud data and a fine standard T-posture training decoder. The human body posture is determined by the shape vector parameters and attitude vector parameters of SMPL. The SMPL parameters are randomly combined to simulate various human body postures, and the human body mesh of the fine standard T-posture is used as the basic true value for training, forming training data for converting any human body posture into a fine standard T-posture.

Images