A three-dimensional human body reconstruction method and system based on a STAR model

Abstract

The application discloses a three-dimensional human body reconstruction method and system based on a STAR model, which uses a convolutional neural network to extract features of input human body information, obtains global features of the human body, and inputs the global features into a time sequence feature extraction network to encode time-related features contained in a video, so that a feature sequence containing time sequence information is obtained, the human body feature sequence is used, a full connection layer is used as a human body parameter regression network to predict SMPL posture parameters and body shape parameters, and a self-attention mechanism is added to improve the stability of an output human body model sequence, posture parameters and body shape parameters of the STAR model are predicted through the human body model parameter regression network, a human body region in the video is taken as input, gender parameters are obtained through a gender detector, and the gender parameters are input into the STAR model to obtain a human body parameter model of a specific gender. The application can effectively express and restore real human body postures and body shapes, and effectively improves reconstruction accuracy and reconstruction speed.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computer vision, and particularly relates to a three-dimensional human body reconstruction method and system based on a STAR model. BACKGROUND

[0002] Virtual reality and 3D display technology are rapidly developing, and new concepts such as "meta universe" and "digital twin" have been proposed. An accurate three-dimensional human body model is particularly necessary for completing a series of tasks such as interaction in the virtual world. However, the traditional way of obtaining a three-dimensional human body model is very limited. One way is to actively model using 3D modeling software and the like, which requires a large amount of human resources and is not efficient. Another way is to obtain 3D information such as depth and point cloud of the target person through devices such as Kinect and laser radar, and then reverse model to obtain a three-dimensional human body model. However, such 3D devices are not common in life, and it is also difficult to obtain related 3D scanning data.

[0003] In the prior art, three-dimensional human body reconstruction algorithms can be mainly divided into two categories. One is a three-dimensional human body reconstruction algorithm based on a geometric method, and the other is a three-dimensional human body reconstruction algorithm based on deep learning. Among them, the method based on deep learning is divided into template-based human body reconstruction and non-template-based human body reconstruction. Based on the geometric method, the image is directly obtained from multiple views by a camera or a depth sensor to reconstruct a 3D scene. In a multi-view environment, the problem of viewpoint association needs to be solved. In addition, the human body is essentially non-rigid, which is a great challenge to the non-rigid registration problem.

[0004] In comparison, the three-dimensional human body reconstruction method based on deep learning has obvious advantages in efficiency. Traditional non-template-based three-dimensional model expression has three types: voxel, point cloud and grid. Voxel is limited by resolution and expression ability and lacks many details; point cloud lacks surface information of an object because there is no connection between points; and grid representation has the characteristics of light weight and rich shape details. In template-based human body reconstruction, one way is to represent the human body by using a skeleton, and researchers have subsequently developed some statistical human body models such as SCAPE (Shape Completion and Animation of People, a deformable human body model based on triangular patches), SMPL (A Skinned Multi-Person Linear Model, a deformable human body model based on point cloud), and the like, to comprehensively reconstruct human body information by adding human body appearance and posture parameters.

[0005] Traditional methods mainly use depth information to reconstruct human body in three dimensions, so the accuracy can reach a very high level. However, processing a large amount of information such as point cloud will cause great pressure on storage, so in recent years many researches on dynamic human body reconstruction are to obtain a human body template first, and then adjust the template, so as to shorten the running time and reduce the memory consumption. However, the existing template-based video pose and shape estimation method usually cannot produce accurate enough prediction results. One of the main reasons behind this is that the temporal features in the video cannot be effectively extracted, and the reconstructed human body model is difficult to capture the complexity and variability of real human motion. SUMMARY

[0006] The main purpose of the present application is to provide a three-dimensional human body reconstruction method and system based on STAR model, aiming at solving the technical problem that the current three-dimensional human body reconstruction method is difficult to capture the complexity and variability of real human motion.

[0007] To achieve the above purpose, the present application provides a three-dimensional human body reconstruction method based on STAR model, which comprises the following steps:

[0008] S1: input a video containing human body information, use a convolutional neural network to extract features of the input human body information of the video, and obtain global features F of the human body;

[0009] S2: input the extracted global features F into a SRU-based time sequence feature extraction network, encode the time-related features contained in the video, and obtain a group of feature sequences G containing time sequence information;

[0010] S3: use the feature sequence G to predict the pose parameters P1 and the body shape parameters P2 of the SMPL model by using a fully connected layer as a human body parameter regression network;

[0011] S4: predict the pose parameters and body shape parameters of the STAR model through the human body model parameter regression network corresponding to the SMPL model;

[0012] S5: input the human body region in the video as input, use a gender detector to obtain gender parameters, input into the STAR model in S04, and output a human body parameter model of a specific gender.

[0013] Optionally, in the step S1, the convolutional neural network adopts a ResNet-50 deep residual network, and when the convolutional neural network is used to extract features of the input human body information of the video, the weights obtained by training the SPIN algorithm are used as the initial weights of the ResNet-50 deep residual network.

[0014] Optionally, the step S1 further comprises adding a YOLOv3-416-bbox human target detector before network feedforward.

[0015] Optionally, in the step S3, the human parameter regression network is composed of two fully connected layers, the feature vector of each frame is input, the initial value of the internal parameter is given, and the predicted SMPL model pose parameter P1 and body shape parameter P2 are obtained through multiple iterations.

[0016] Optionally, in the step S3, when performing iterative optimization, the predicted parameter is initialized as an average pose, and then input to the human parameter regression network.

[0017] Optionally, the step S4 specifically comprises:

[0018] The vertex offset caused by the body shape parameter and the pose parameter is calculated in turn, and then superimposed to obtain the average human mesh in T-pose, and finally the human mesh model vertex is calculated through skinning operation;

[0019] Based on the multi-target loss function of human three-dimensional reconstruction, the STAR model is obtained by error feedback iterative prediction.

[0020] Optionally, the expression of the human mesh model vertex is specifically:

[0021] M(β,θ)=M(T p (β,θ),J(β),θ,W)

[0022] Wherein,

[0023]

[0024] is the average shape, B s (β) is the vertex offset caused by the body shape parameter, B p (q,β2) is the vertex offset caused by the pose parameter.

[0025] Optionally, the expression of the multi-target loss function based on human three-dimensional reconstruction is specifically:

[0026] L=L STAR +L cam +L 2d +L 3d

[0027] Wherein, L STAR is the STAR model parameter loss, L cam is the camera parameter loss, L 2d is the 2D joint loss, and L 3d is the 3D joint loss.

[0028] Optionally, in step S5, the gender detector comprises two parts: one part is VGG-16, which serves as a feature extraction network to extract gender features, and the other part is a Softmax layer, which serves as a classifier to distinguish between males and females.

[0029] Furthermore, to achieve the above objectives, the present invention also provides a three-dimensional human body reconstruction system based on the STAR model, wherein the three-dimensional human body reconstruction device based on the STAR model includes:

[0030] Video feature extraction module: Input a video containing human information, use a convolutional neural network to extract features from the human information in the input video, and obtain the global features F of the human body;

[0031] Temporal feature extraction module: The extracted global features F are input into the SRU-based temporal feature extraction network to encode the time-related features contained in the video, resulting in a set of feature sequences G containing temporal information;

[0032] Spatial position prediction module: Using the feature sequence G, a fully connected layer is used as a human parameter regression network to predict the pose parameter P1 and body shape parameter P2 of the SMPL model;

[0033] STAR model generation module: predicts the pose and body shape parameters of the STAR model through the regression network of the human body model parameters corresponding to the SMPL model;

[0034] Gender feature determination module: Taking the human body region in the video as input, the gender detector obtains gender parameters, which are then input into the STAR model described in S04, and output a human body parameter model for a specific gender.

[0035] This invention proposes a 3D human body reconstruction method and system based on the STAR model. The method utilizes a convolutional neural network to extract features from the input human body information, obtaining global human body features, which are then input into a temporal feature extraction network. Temporally relevant features contained in the video are encoded to obtain a set of feature sequences containing temporal information. Using these human body feature sequences, a fully connected layer is used as a human body parameter regression network to predict the pose and body shape parameters of the SMPL (Simultaneous Mental Phenomenon Scale). A self-attention mechanism is added to improve the stability of the output human body model sequence. The pose and body shape parameters of the STAR model are predicted through the human body model parameter regression network. Using the human body region in the video as input, a gender detector obtains gender parameters, which are then input into the STAR model to obtain a human body parameter model for a specific gender. This invention can effectively express and restore realistic human body posture and body shape, significantly improving reconstruction accuracy and speed. Attached Figure Description

[0036] Figure 1 This is a flowchart illustrating a three-dimensional human body reconstruction method based on the STAR model in this invention.

[0037] Figure 2 This is a schematic diagram illustrating the principle of the three-dimensional human body reconstruction method based on the STAR model in this invention;

[0038] Figure 3 This is a schematic diagram of the structure of the global human appearance feature extraction network in this invention;

[0039] Figure 4 This is a schematic diagram of the structure of the human motion temporal feature extraction network in this invention;

[0040] Figure 5 This is a schematic diagram of the SMPL human body model parameter regression network structure in this invention;

[0041] Figure 6 This is a schematic diagram of the STAR human body model input parameter processing flow in this invention.

[0042] The realization of the objective, functional features and advantages of the present invention will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation

[0043] It should be understood that the specific embodiments described herein are merely illustrative of the invention and are not intended to limit the invention.

[0044] This invention provides a three-dimensional human body reconstruction method based on the STAR model, referring to... Figure 1 , Figure 1 This is a flowchart illustrating an embodiment of the three-dimensional human body reconstruction method based on the STAR model of the present invention.

[0045] In this embodiment, the three-dimensional human body reconstruction method based on the STAR model includes the following steps:

[0046] Step 1: Video Feature Extraction: Input a video containing human information, and use a convolutional neural network to extract features from the input human information to obtain the global features F of the human body;

[0047] Step 1-1: Before the network feedforward, add a YOLOv3-416-bbox human target detector to make the algorithm as fast as possible while ensuring the accuracy of human reconstruction.

[0048] Steps 1-2: Use the weights obtained from training with the SPIN algorithm as the initial weights of ResNet-50. Since the SPIN algorithm has learned prior knowledge about the human body through training, it helps to improve the model performance. Then, ResNet-50 with a deep residual network is used to extract the global feature vector of the human body.

[0049] Step 2: Temporal Feature Extraction: The extracted global features F are input into a temporal feature extraction network based on SRU (Simple RecurrentUnit) to encode the time-related features contained in the video, resulting in a set of feature sequences G containing temporal information;

[0050] Step 3: Spatial position prediction: Using the human feature sequence G, a fully connected layer is used as a human parameter regression network to predict the pose parameters P1 and body shape parameters P2 of SMPL, and a self-attention mechanism is added to improve the stability of the output human model sequence by changing the degree of attention to feature vectors in different frames through the use of weights.

[0051] Step 3-1: The human model parameter regression network consists of two fully connected layers. The input is the feature vector of each frame. Given the initial values ​​of the internal parameters, the predicted SMPL parameters are obtained after multiple iterations of optimization.

[0052] Step 3-2: Initialize the extracted prediction parameters to the average attitude θ. mean Then, the data is input into the parameter regression network to extract the motion details lost in a single frame image.

[0053] Step 4: STAR Model Generation: Predict the pose and body shape parameters of the STAR model using the SMPL human model parameter regression network;

[0054] It's important to note that STAR (Sparse Trained Articulated Human BodyRegressor) is an alternative to SMPL. It's a compact, realistic human body model that learns a locally correctable human shape model in sparse space. Compared to SMPL, it has fewer internal parameters, is more lightweight, and has greater expressive power. STAR improves upon SMPL in five main aspects.

[0055] 1. The model's internal parameters are reduced to 20% of SMPL.

[0056] 2. The changes in the human body model caused by changes in Pose parameters are sparse and local, meaning that the movement of the left wrist will not affect the shape of the right leg.

[0057] 3. The model's derivatives are sparse because the parameters are sparse and local.

[0058] 4. The number of training human models increased from 4,000 to 14,000.

[0059] 5. The changes in the human body model resulting from changes in pose parameters should be related to body shape. That is, the shape changes in the legs of a fat person and a thin person should be different.

[0060] Step 4-1: Similar to the SMPL human body model, the STAR human body model has a predefined template human body mesh T∈R with a total of N = 6890 vertices. 3N The vertex offsets caused by body shape parameters and pose parameters are calculated sequentially, then superimposed to obtain the average human body mesh under T-pose. Finally, the vertices of the human body mesh model are calculated through skinning operations. The specific calculations are as follows:

[0061] (1) Calculate the vertex offset caused by the body size parameters:

[0062] Shape blending deformation function B S (β;S):R |β| →R 3N Calculate the vertices of the human body mesh based on body shape parameters:

[0063]

[0064] Where β = [β1, β2, ..., β] |β| [] represents body shape parameters, S = [S1, S2…S] |β| ]∈R 3N×|β| To capture the PCA basis for human body shape variations, body shape offsets were then superimposed onto the average shape. superior:

[0065]

[0066] (2) Calculate the vertex offset caused by the attitude parameters:

[0067] In STAR's human body model, the output of the body shape and pose deformation function is further deformed by the pose blending deformation function. The vertex offset caused by the pose parameters is conditional on the human's pose and body shape, making the LBS-calculated human body mesh look more realistic after adding pose and body shape correction offsets. Therefore, STAR's pose blending is defined as follows:

[0068]

[0069] Where K j q is the regression matrix of the attitude correction offset at the j-th joint. ne(j) This represents the quaternion values ​​of the j-th joint and its adjacent nodes, β2 is the second body size parameter, and A j It is the vertex weight of the learning.

[0070] (3) Skin

[0071] After the above two steps of body shape blending deformation and posture blending deformation, the final human body mesh vertices are obtained by calculating the skinning function W:

[0072] M(β,θ)=M(T p (β,θ),J(β),θ,W)

[0073] Among them, T p The formula for calculating (β,θ) is:

[0074]

[0075] in, For average shape, B S (β) represents the vertex offset caused by the body size parameter, B p (q,β2) represents the vertex offset caused by the attitude parameters.

[0076] Step 4-2: Use an error feedback iterative process to more accurately predict the parameterized human body model. Since the accuracy of the output human body model is primarily measured by the distance error between the output network vertex positions and the real positions, this invention selects the L2 loss function based on distance metric and designs a multi-objective loss function for human 3D reconstruction, comprising four parts: STAR parameter loss L... STAR Camera parameter loss L cam 2D joint loss L 2d 3D joint loss L 3d Therefore, the calculation formula is as follows:

[0077] L = L STAR +L cam +L 2d +L 3d

[0078] Step 5: Gender Feature Determination: Taking the human body region in the video as input, a gender detector is used to obtain gender parameters, which are then input into the STAR model described in Step 4. The output is a human body parameter model for the specific gender. The gender detector consists of two parts: a VGG-16 layer, which acts as a feature extraction network to extract gender features, and a Softmax layer, which acts as a classifier to distinguish between males and females, thus obtaining the gender parameters of the target person, which are then input into the STAR human body model described in Step 4.

[0079] This embodiment provides a 3D human body reconstruction method based on the STAR model. The video feature extraction module and temporal feature extraction module designed in this application can effectively extract the pose and body shape information of the target human body compared with existing lightweight network modules. The SRU module included in this application, compared with the existing GRU module, removes the dependency of calculation between previous and next time steps, making it easier to train and greatly improving training efficiency. In addition, a soft attention mechanism is added after the temporal feature extraction module. The scoring function expresses the correlation between different feature vectors. Compared with the hard attention mechanism, it is differentiable and the parameters of the scoring function can be learned through backpropagation.

[0080] Based on this, the spatial location prediction module in this embodiment uses the structured model STAR as the human body model output. Combined with the gender parameters obtained by the gender detector trained with the existing dataset, this lightweight model can effectively express and restore the real human posture and body shape, and perform more efficient human body 3D reconstruction under limited computing resources.

[0081] To explain this application more clearly, a specific example of a three-dimensional human body reconstruction method based on the STAR model is presented.

[0082] like Figure 2 As shown, the steps of the three-dimensional human body reconstruction method based on the STAR model in this embodiment include:

[0083] Step 1: Video Feature Extraction. This embodiment specifically utilizes a ResNet-50 residual network with deep residual networks to extract features from a given surveillance video of length n. Global features are extracted from each frame of the input video. The initial weights of ResNet-50 are obtained by training with the SPIN algorithm. Since the SPIN algorithm has learned prior knowledge of human beings through training, it helps to improve the performance of the model.

[0084] Before the network feedforward, this embodiment also uses YOLOv3-416-bbox as a human target detector to make the algorithm as fast as possible while ensuring the accuracy of human reconstruction.

[0085] like Figure 3As shown, in this embodiment, each frame of the video input is uniformly 224×224 through a resize operation. First, a 7×7 convolution with a stride of 2 is used to increase the feature map channels to 64. Then, the first residual structure quadruples the number of channels, followed by a second residual structure that transforms the feature map into (28, 28, 512). The next three residual structures successively reduce the feature map size and increase its dimensionality, resulting in (14, 14, 1024), (7, 7, 2048), and (1, 1, 2048) channels respectively. Finally, it is expanded into a 2048-channel feature vector. The residual structures in ResNet allow for deeper networks, faster convergence during training, and a smaller model size compared to previous deep feature extraction networks. This addresses issues such as degradation, gradient explosion, and gradient vanishing in deep networks.

[0086] When the network depth exceeds 50 layers, ResNet begins to adopt the Bottleneck structure to introduce 1×1 convolutions. While 1×1 convolutions significantly reduce network computational complexity compared to other convolutional sizes—for example, stacking two 3×3 convolutions would only result in one ReLU—using 1×1 convolutions can lead to two ReLUs, thus increasing the number of non-linear computation layers. Furthermore, excessively deep networks can result in overly large models and wasted resources. Therefore, this invention uses ResNet-50 with a depth of 50 as the global human appearance feature extraction network. Table 1 shows ResNets of different depths.

[0087] Table 1 ResNets of different depths

[0088]

[0089]

[0090] Step 2: Temporal feature extraction. For example... Figure 4 As shown, in this embodiment, the input video frames f1, f2, f3, ... f are processed by ResNet-50 to extract features, and then encoded into time-related features g1, g2, g3, ..., g through a temporal feature extraction network containing SRU (Simple Recurrent Unit). N The sequence. Compared with GRU, the SRU structure removes the dependency on calculations at previous and next time steps, making it easier to train and greatly improving training efficiency.

[0091] In the basic SRU structure, f represents the forget gate, r represents the reset gate, h represents the output state, and c represents the internal state. The complete SRU architecture can be decomposed into two sub-components: a lightweight recurrent network and a high-speed network. The SRU calculation formula can be expressed as follows:

[0092] First, the lightweight loop component continuously reads the input vector x. t And calculate the state sequence c of the captured information. t This process is similar to LSTM and GRU algorithms, with a forget gate f t It can control the flow of information. t It can be represented as:

[0093] f t =σ(W f x t +v f ⊙c t-1 +b f )

[0094] According to f t Adaptive average previous state c t-1 and current observations Determine the state vector c t :

[0095]

[0096] Reset door r t It can be represented as:

[0097] r t =σ(W r x t +v r ⊙c t-1 +b r )

[0098] Secondly, the high-speed network component facilitates gradient-based deep network training, which uses a reset gate r t Adaptive combined input x t The state c generated by the aforementioned lightweight loop t , where (1-r t )⊙x t This is a skip connection that allows gradients to propagate directly to the previous layer, improving scalability. The combination of these two components results in a simple and efficient overall architecture, enabling parallel processing. Its output state h t It can be represented as:

[0099] h t =r t ⊙c t +(1-r t )⊙x t

[0100] In the above formula, W, W f W r It is a parameter matrix, v f v r bf b r These are parameter vectors, all of which need to be learned during training.

[0101] Step 3: Spatial location prediction. For example... Figure 5 As shown, the human model parameter regression network consists of two fully connected layers, each with 1024 neurons. Its input is the feature vector of each frame, given initial values ​​for the internal parameters. The predicted SMPL parameters are obtained through k iterations of optimization. These internal parameters include pose parameters, body shape parameters, and camera parameters. The human parameter regressor uses initial pose parameters and body shape parameters θ. t And the visual temporal features of a single frame g t To calculate the estimate θ after k iterations t ′. Where the initial attitude parameter is the average attitude θ. mean Instead of setting it directly to 0, this allows the parameter regressor to combine global and temporal features of the image to improve human motion and pose estimation, while also augmenting the human motion details lost in a single frame at the frame level.

[0102] In the aforementioned spatial location prediction step, the pose parameter P1 among the predicted SMPL parameters undergoes processing via a soft attention mechanism. By learning the contribution of each video frame and amplifying the contribution of the most important frame, a unique SMPL-β body shape parameter can be obtained, ensuring consistency in the human body model sequence and improving the stability of the predicted human body model sequence. The essence of the self-attention mechanism is a weighted processing of the input information. Assuming there are n m-dimensional vectors... By using a self-attention mechanism to weight and integrate these n vectors, we can obtain a weighted vector x. * As shown in the following formula:

[0103]

[0104] Where, α i These are the attention weights, for the more important vector x. i Give a larger weight α i Unlike hard attention mechanisms, soft attention mechanisms assign weights to each location of a feature, using these weights to represent the level of attention given to a region. Soft attention mechanisms combine the correlation between different feature vectors through a scoring function, and because they are differentiable, the parameters of the scoring function can be learned through backpropagation, allowing the weight parameters to be obtained through network training.

[0105] f i =f(g i )

[0106]

[0107]

[0108] Where the weight α i The model learns through an MLP f-learning algorithm, then uses Softmax for normalization to form a probability distribution, and finally generates the body shape parameter β from the human body parameter regression network. i The product of the product with the corresponding weights yields a unified body shape parameter β.

[0109] Step 4: STAR Model Generation. This invention introduces the sparsely trained jointed human regressor STAR human model as the network's output model. Each joint of STAR only affects the vertices around the joint. This sparse representation can generate more realistic pose and body shape deformations, and reduces the number of model parameters to 20% of the SMPL human model, while also providing richer expressive power. Although the SMPL human model and the STAR human model are similar, the pose parameter θ∈R... 72 Used to represent human posture, body shape parameter β∈R 10 SMPL parameters are used to represent human body shapes, but because they differ significantly in pose and shape spaces, they cannot be directly input into the STAR model. Therefore, the STAR model needs to be iteratively optimized from scratch based on the SMPL model before being input into the STAR human body model for calculation.

[0110] like Figure 6 As shown, the SMPL parameter labels contain 72 SMPL-θ pose parameters, 10 SMPL-β body shape parameters, and 3 ε camera parameters. The pose parameter θ includes the global body rotation expressed in axis-angle form and the relative rotation of 23 joints. The body shape parameter β consists of the first 10 coefficients in the PCA shape space. Similar to the SMPL human model, the STAR human model begins with a predefined template human mesh T∈R with N = 6890 vertices in a T-pose state. 3N The human kinematics tree has K=24 joints. Next, the vertex offsets caused by body shape parameters and pose parameters are calculated sequentially, and then superimposed to obtain the average human mesh under T-pose. Finally, the vertices of the human mesh model are calculated through skinning operations. The specific calculations are as follows:

[0111] (1) Calculate the vertex offset caused by the body size parameters:

[0112] Shape blending deformation function B S (β;S):R |β| →R 3N Calculate the vertices of the human body mesh based on body shape parameters:

[0113]

[0114] Where β = [β1, β2, ..., β] |β| [] represents body shape parameters, S = [S1, S2…S] |β| ]∈R 3N×|β| To capture the PCA basis for human body shape variations, body shape offsets were then superimposed onto the average shape. superior:

[0115]

[0116] (2) Calculate the vertex offset caused by the attitude parameters:

[0117] In STAR's human body model, the output of the body shape and pose deformation function is further deformed by the pose blending deformation function. The vertex offset caused by the pose parameters is conditional on the human's pose and body shape, making the LBS-calculated human body mesh look more realistic after adding pose and body shape correction offsets. Therefore, STAR's pose blending is defined as follows:

[0118]

[0119] Where K j q is the regression matrix of the attitude correction offset at the j-th joint. ne(j) This represents the quaternion values ​​of the j-th joint and its adjacent nodes, β2 is the second body size parameter, and A j It is the vertex weights learned.

[0120] (3) Skin

[0121] After the above two steps of body shape blending deformation and posture blending deformation, the final human body mesh vertices are obtained by calculating the skinning function W:

[0122] M(β,θ)=M(T p (β,θ),J(β),θ,W)

[0123] Among them, T p The formula for calculating (β,θ) is:

[0124]

[0125] The final STAR model output by this invention, like the SMPL model, is also a differentiable function M(θ,β)∈R. 6890 The output is a 3D model in mesh form containing 6890 vertices.

[0126] The error feedback iterative process is used to more accurately predict the parameterized human model. Since the accuracy of the output human model is primarily measured by the distance error between the output network vertex positions and the ground truth positions, this invention selects the distance-based L2 loss function and designs a multi-objective loss function for human 3D reconstruction, comprising four parts: STAR parameter loss L... STAR Camera parameter loss L cam 2D joint loss L 2d 3D joint loss L 3d Therefore, the calculation formula is as follows:

[0127] L = L STAR +L cam +L 2d +L 3d

[0128] L STAR =ω β L β +ω θ L θ

[0129]

[0130] The STAR parameter loss function consists of two parts: body size parameter L. β Loss term and attitude parameter L θ Loss term, ω β and ω θ These are body size parameters L β Loss term and attitude parameter L θ Weights of the loss term. Body size parameter L. β The loss term is the prediction of a single body size parameter. L2 loss, attitude parameter L θ The loss term is the sum of the predictions from all frames. L2 loss.

[0131] The loss function for camera parameters is shown in the following equation, where L cam It is the camera parameter loss term, ω ε L represents the weight of the camera loss term. ε It is calculated from the L2 loss of the predicted value and the label value.

[0132] L cam =ω ε L ε

[0133]

[0134] Since the model predicts camera parameters, and the STAR model can calculate the 3D joints of the human body model based on the input parameters, the calculation formula predicted by the algorithm based on the camera imaging principle is as follows, where J p2d J represents the coordinates of 2D human joints. p3d The coordinates of the 3D joints of the human body calculated for the STAR human body model, where s is the scaling factor and t is the translation vector.

[0135] J p2d =sJ p3d +t

[0136] To ensure that the generated 3D human body model matches the pose of the real target human body, real human 2D joint point labels are needed for supervision. Therefore, a 2D joint point loss function is introduced, as shown in the following formula:

[0137] L 2d =ω j2d L j2d

[0138] Where, ω j2d It is the loss weight of 2D human body joints, L j2d The 2D joint loss term is calculated as follows:

[0139]

[0140] Where x n These are the label values ​​for 2D joints. These are the predicted values ​​for 2D joints.

[0141] Because images may become distorted or ambiguous when mapped from 2D to 3D in monocular vision, human body 3D joint loss was also used.

[0142] L 3d =ω j3d L j3d

[0143] Among them, L 3d For the loss term of the 3D human body joints, ω j3d For the loss weights of 3D human body joints, L j3d This represents the loss value of the 3D joints of the human body.

[0144]

[0145] in To predict the coordinates of the generated 3D human body joints, X n The coordinates of the 3D joints of the human body in the label.

[0146] Step 5: Gender Feature Determination. STAR human models are categorized into male, female, and gender-neutral models. While most current methods use the gender-neutral model as output, the difference in body shape between men and women may affect the prediction accuracy of the resulting human model. This invention adds a gender detection network before STAR model calculation. The target human body region in the video is used as input, and the gender of the target human body is determined frame by frame by the gender detection network. The human model parameters predicted by the human model parameter regression network are then input into the STAR-male and STAR-female models, respectively. After STAR model skinning calculation, a 3D human model is obtained. This allows the system to learn the gender of the person based on the input video and adaptively output model parameters specific to that person.

[0147] The gender detection network first uses VGG-16 as a feature extraction network to extract gender features, then a Softmax layer is used as a classifier to distinguish between males and females, and finally the gender parameters of the target person are input into the STAR human body model.

[0148] In a preferred embodiment, a three-dimensional human body reconstruction device based on the STAR model is also proposed, specifically including:

[0149] Video feature extraction module: Input a video containing human information, use a convolutional neural network to extract features from the human information in the input video, and obtain the global features F of the human body;

[0150] Temporal feature extraction module: The extracted global features F are input into the SRU-based temporal feature extraction network to encode the time-related features contained in the video, resulting in a set of feature sequences G containing temporal information;

[0151] Spatial position prediction module: Using the feature sequence G, a fully connected layer is used as a human parameter regression network to predict the pose parameter P1 and body shape parameter P2 of the SMPL model;

[0152] STAR model generation module: predicts the pose and body shape parameters of the STAR model through the regression network of the human body model parameters corresponding to the SMPL model;

[0153] Gender feature determination module: Taking the human body region in the video as input, the gender detector obtains gender parameters, which are then input into the STAR model described in S04, and output a human body parameter model for a specific gender.

[0154] Other embodiments or specific implementations of the STAR model-based three-dimensional human body reconstruction device of the present invention can be referred to the above-described method embodiments, and will not be repeated here.

[0155] Furthermore, this invention also proposes a storage medium storing a program for a three-dimensional human body reconstruction method based on a STAR model. When executed by a processor, the program implements the steps of the three-dimensional human body reconstruction method based on a STAR model as described above. Therefore, it will not be repeated here. Additionally, the beneficial effects of using the same method will not be repeated. For technical details not disclosed in the computer-readable storage medium embodiments of this application, please refer to the description of the method embodiments of this application. As an example, program instructions can be deployed to execute on a single computing device, or on multiple computing devices located in one location, or on multiple computing devices distributed across multiple locations and interconnected via a communication network.

[0156] Those skilled in the art will understand that all or part of the processes in the above embodiments can be implemented by a computer program instructing related hardware. The program can be stored in a computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. The storage medium can be a magnetic disk, optical disk, read-only memory (ROM), or random access memory (RAM), etc.

[0157] Through the above description of the embodiments, those skilled in the art can clearly understand that the present invention can be implemented by means of software plus necessary general-purpose hardware, and of course, it can also be implemented by special hardware including application-specific integrated circuits, special CPUs, special memory, special components, etc. Generally, any function performed by a computer program can be easily implemented by corresponding hardware, and the specific hardware structure used to implement the same function can also be diverse, such as analog circuits, digital circuits, or special-purpose circuits. However, for the present invention, software program implementation is more often the preferred implementation method. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a readable storage medium, such as a computer floppy disk, USB flash drive, portable hard disk, read-only memory (ROM), random access memory (RAM), magnetic disk, or optical disk, etc., including several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments of the present invention.

Claims

1. A three-dimensional human body reconstruction method based on the STAR model, characterized in that, The method includes the following steps: S1: Input a video containing human information, and use a convolutional neural network to extract features from the human information in the input video to obtain the global features F of the human body; S2: Input the extracted global features F into the SRU-based temporal feature extraction network to encode the time-related features contained in the video, and obtain a set of feature sequences G containing temporal information; S3: Using the feature sequence G, the fully connected layer is used as a human body parameter regression network to predict the pose parameter P1 and body shape parameter P2 of the SMPL model; S4: Predict the pose and body shape parameters of the STAR model using the regression network of the human body model parameters corresponding to the SMPL model; S5: Using the human body region in the video as input, the gender parameter is obtained using a gender detector, and then input into the STAR model described in S04 to output a human body parameter model for a specific gender.

2. The three-dimensional human body reconstruction method based on the STAR model as described in claim 1, characterized in that, In step S1, the convolutional neural network adopts a ResNet-50 deep residual network. When using the convolutional neural network to extract features of human information from the input video, the weights obtained by training the SPIN algorithm are used as the initial weights of the ResNet-50 deep residual network.

3. The three-dimensional human body reconstruction method based on the STAR model as described in claim 1, characterized in that, Step S1 also includes adding a YOLOv3-416-bbox human target detector before the network feedforward.

4. The three-dimensional human body reconstruction method based on the STAR model as described in claim 1, characterized in that, In step S3, the human body parameter regression network consists of two fully connected layers. The feature vector of each frame is input, and the initial values ​​of the internal parameters are given. After multiple iterations and optimizations, the pose parameters P1 and body shape parameters P2 of the predicted SMPL model are obtained.

5. The three-dimensional human body reconstruction method based on the STAR model as described in claim 1, characterized in that, In step S3, during iterative optimization, the extracted prediction parameters are initialized as the average pose and then input into the human parameter regression network.

6. The three-dimensional human body reconstruction method based on the STAR model as described in claim 1, characterized in that, Step S4 specifically includes: The vertex offsets caused by body shape parameters and pose parameters are calculated sequentially, and then superimposed to obtain the average human body mesh under T-pose. Finally, the vertices of the human body mesh model are calculated through skinning operations. The STAR model is obtained by using a multi-objective loss function based on human 3D reconstruction and error feedback iterative prediction.

7. The three-dimensional human body reconstruction method based on the STAR model as described in claim 6, characterized in that, The expression for the vertices of the human body mesh model is as follows: M(β,θ)=M(T p (β,θ),J(β),θ,W) in, For average shape, B S (β) represents the vertex offset caused by the body size parameter, B p (q,β2) represents the vertex offset caused by the attitude parameters.

8. The three-dimensional human body reconstruction method based on the STAR model as described in claim 6, characterized in that, The expression for the multi-objective loss function based on human 3D reconstruction is as follows: L=L STAR +L cam +L 2d +L 3d Among them, L STAR For the STAR model parameter loss, L cam For camera parameter loss, L 2d For 2D joint loss, L 3d This refers to the loss of 3D joints.

9. The three-dimensional human body reconstruction method based on the STAR model as described in claim 1, characterized in that, In step S5, the gender detector consists of two parts: one part is VGG-16, which is used as a feature extraction network to extract gender features, and the other part is a Softmax layer, which is used as a classifier to distinguish between males and females.

10. A three-dimensional human body reconstruction system based on the STAR model, characterized in that, The STAR model-based three-dimensional human body reconstruction device includes: Video feature extraction module: Input a video containing human information, use a convolutional neural network to extract features from the human information in the input video, and obtain the global features F of the human body; Temporal feature extraction module: The extracted global features F are input into the SRU-based temporal feature extraction network to encode the time-related features contained in the video, resulting in a set of feature sequences G containing temporal information; Spatial position prediction module: Using the feature sequence G, a fully connected layer is used as a human parameter regression network to predict the pose parameter P1 and body shape parameter P2 of the SMPL model; STAR model generation module: predicts the pose and body shape parameters of the STAR model through the regression network of the human body model parameters corresponding to the SMPL model; Gender feature determination module: Taking the human body region in the video as input, the gender detector obtains gender parameters, which are then input into the STAR model described in S04, and output a human body parameter model for a specific gender.

Images