A method and system for reconstructing a full-body human model based on spiral graph convolutional network

Abstract

The present invention belongs to the field of artificial intelligence technology, and specifically relates to a full-body human body model reconstruction method and system based on a spiral graph convolutional network, and a corresponding full-body human body model reconstruction device. The full-body human body model reconstruction method includes the following steps: S1: extracting local images of the whole body, hand joints and face from the original image. S2: extracting the feature vectors corresponding to each local image through the feature extraction network. S3: using the spiral graph convolutional network to perform graph structure processing and predict the position of each vertex to reconstruct the corresponding local model. S4: integrating the reconstructed body model, hand model, and face model according to the defined vertex sequence to obtain the required three-dimensional full-body model including facial expressions and gestures. The present invention solves the problems of various existing schemes in reconstructing three-dimensional human body models, such as irregular and non-smooth model surfaces, insufficient feature information of the face and hands, and poor overall performance of the model.

Description

Technical Field

[0001] The present invention belongs to the field of artificial intelligence technology, and specifically relates to a whole-body human model reconstruction method and system based on a spiral graph convolutional network, and corresponding whole-body human model reconstruction equipment. Background Art

[0002] Accurately and rapidly reconstructing a full-body 3D human model from a single image has become a key research topic in computer vision in recent years. This task involves simultaneously predicting the various components of a 3D human mesh model, including the body, face, and hands, from a single RGB image. Reconstructing a full-body model can help us more easily understand human behavior and psychological activities. Motion capture, an application of machine vision, is also a foundational technology in artificial intelligence robotics research.

[0003] However, due to the diverse poses people typically assume in image space, the variability of image backgrounds due to changes in their surroundings, and the tendency for people to be obscured by their companions or other objects, 2D images may not capture all the features required for 3D model generation, making 3D modeling based on 2D images extremely challenging. Furthermore, compared to the entire body, the face and hands occupy a very small proportion of a 2D image. Furthermore, the complex articulation of the hand and the high degree of deformation of facial muscles make it even more difficult to accurately reconstruct facial expressions and gestures.

[0004] For these reasons, most previous 3D modeling work based on 2D images has independently studied the reconstruction of the body, face, and hands, with relatively little research on full-body reconstruction. Existing full-body 3D human mesh reconstruction methods are divided into three modules, each corresponding to a sub-network for reconstructing the body, face, and hands, respectively. Most of these methods are based on parametric human models, such as SMPL-X, Adam, and GHUM / GHUML. Parameters such as posture and body shape are predicted through the corresponding network and then input into the corresponding parametric human model layer, ultimately resulting in a 3D human mesh. Expose is a method that uses SMPL-X to reconstruct a full-body model from RGB images.

[0005] However, it is not easy to predict parameters to complete full body reconstruction. In order to make the reconstructed human body model accurate and vivid, it is necessary to predict multiple different posture parameters. In the parametric model space, posture parameters can be expressed in various forms, such as axis angles, rotation matrices, quaternions, etc. Each rotation expression also has its own limitations, which increases the complexity of network prediction.

[0006] In the latest research, in order to avoid the complexity caused by directly predicting model parameters, the reconstruction process can be regarded as predicting the coordinate position of each vertex in the human mesh model, which corresponds to the model-free non-parametric method. Since the topological shape of the human mesh model can be naturally regarded as a graph structure, graph convolutional neural networks have attracted widespread attention in this type of method due to their ability to effectively extract local spatial features and explicitly encode the graph structure. Although methods based on graph convolutional neural networks are well suited for human mesh topology, most current research focuses on accurately estimating body posture, and few works use graph convolutional networks to reconstruct expressive full-body human models. In addition, current methods based on graph convolutional networks also have the problem of irregular and non-smooth surface shapes of predicted human models. Summary of the Invention

[0007] In order to solve the problems of irregular and non-smooth model surface, insufficient feature information of face and hands, and poor overall performance of the model when reconstructing the three-dimensional human body model in various existing schemes, the present invention provides a full-body human body model reconstruction method and system based on spiral graph convolutional network, and corresponding full-body human body model reconstruction equipment.

[0008] The present invention is achieved by adopting the following technical solutions:

[0009] A full-body model reconstruction method based on a spiral graph convolutional network is used to reconstruct a 3D full-body model of the corresponding person, including facial expressions and gestures, based on a 2D image of the person. The full-body model reconstruction method includes the following steps:

[0010] S1: Preprocess the input person image to extract the partial images corresponding to the human body, hand joints, and face respectively; and adjust the size of the obtained body image, hand image, and face image to 224*224.

[0011] S2: Remove the fully connected layer from the ResNet50 convolutional neural network and use the average pooling layer output as the final output, which is then used as the required feature extraction network. The pre-trained feature extraction network is used to extract features from the body image, hand image, and face image, respectively, to obtain the corresponding 2048-dimensional feature vector F. b 、F h 、F f .

[0012] S3: Based on the reference topology of each part of the human body and the extracted feature vectors, a spiral graph convolutional network is used to process the graph structure and predict the position of each vertex to reconstruct the corresponding local model. The specific process is as follows:

[0013] S31: The corresponding parameterized grid model is used as the reference topology structure of the spiral graph convolutional network.

[0014] S32: embedding the feature vectors extracted from each local image into each vertex of the parameterized network model to obtain vertex features containing image feature information.

[0015] S33: Input the vertex features into the spiral graph convolutional network, transfer information between vertices in the form of spiral convolution, and finally output the predicted position of each vertex.

[0016] S34: Based on the vertex positions predicted by the spiral graph convolutional network, the corresponding 3D local models are generated in the parameterized mesh model; including the body model, hand model and face model.

[0017] S4: Integrate the reconstructed body model, hand model, and face model according to the vertex sequence defined in the SMPL-X parameterized human mesh model to obtain the required three-dimensional full-body model including facial expressions and gestures.

[0018] As a further improvement of the present invention, in step S1, the process of extracting local images corresponding to different parts based on the person image is as follows:

[0019] S11: performing target detection on the input person image using a human body detection algorithm, a joint detection algorithm, or a face detection algorithm.

[0020] S12: Sort the pixels of the identified target area by coordinate value to obtain the maximum value x of the horizontal coordinate max and the minimum value x min , and the maximum value of the ordinate y max and the minimum value y min ; Then determine the center coordinates of the detection box of each target area and the cropping scale:

[0021]

[0022] S13: According to the center coordinates of the detection frame corresponding to different parts of the person and the cropping scale, the cropped local images I are obtained through the affine transformation function. b for:

[0023] I b =(I; center, scale),

[0024] Where I represents the original input person image.

[0025] S14: setting the size of each cropped local image to 224*224.

[0026] As a further improvement of the present invention, in step S3, the body model is reconstructed as follows:

[0027] S31A: The SMPL-X parameterized human mesh model is used as the reference topology for the spiral graph convolutional network.

[0028] S32A: The feature vector F of the body image b Embedded into the vertices of the SMPL-X human body mesh model to obtain vertex features containing image feature information.

[0029] S33A: Input vertex features into the spiral graph convolutional network, transfer information between vertices in the form of spiral convolution, and finally output the predicted position of each vertex.

[0030] S34A: Generate a corresponding body model in the SMPL-X parameterized human mesh model based on the vertex positions predicted by the spiral graph convolutional network.

[0031] As a further improvement of the present invention, in step S3, the hand model is reconstructed as follows:

[0032] S31B: The MANO parameterized hand mesh model is used as the reference topology for the spiral graph convolutional network.

[0033] S32B: The feature vector F of the facial image h Embedded into the vertices of the MANO parameterized hand mesh model, vertex features containing image feature information are obtained.

[0034] S33B: Input vertex features into the spiral graph convolutional network, transfer information between vertices in the form of spiral convolution, and finally output the predicted position of each vertex.

[0035] S34B: Generate a corresponding hand model in the MANO parameterized hand mesh model based on the vertex positions predicted by the spiral graph convolutional network.

[0036] As a further improvement of the present invention, in step S3, the reconstruction process of the face model is as follows:

[0037] S31C: The FLAME parameterized face mesh model is used as the reference topology for the spiral graph convolutional network.

[0038] S32C: The feature vector F of the facial image f Embedded into the vertices of the FLAME parameterized face mesh model to obtain vertex features containing image feature information.

[0039] S33C: Input vertex features into the spiral graph convolutional network, transfer information between vertices in the form of spiral convolution, and finally output the predicted position of each vertex.

[0040] S34C: Generates a corresponding face model in the FLAME parameterized face mesh model based on the vertex positions predicted by the spiral graph convolutional network.

[0041] As a further improvement of the present invention, in the spiral graph convolutional network of step S3, based on the central vertex in the given graph structure, the aggregation order of adjacent vertex features is constructed in a spiral manner to obtain the corresponding spiral sequence S(v,n):

[0042]

[0043] In the above formula, the spiral sequence S(v,n) is an ordered set of n vertices v in k rings; 0-ring(v) = {v} represents the zeroth ring around the central vertex v in a given graph structure, and the zeroth ring is the central vertex v; k-disk(v) = ∪ i＝0，1，...，k i-ring(v) represents the k-th disk surface surrounding the central vertex v. The k-th disk consists of the union of the 0th ring, the 1st ring, and the k-th ring. N(k-ring(v)) represents the set of adjacent vertices of all vertices in the k-th ring. (k+1)-ring(v) represents the k+1th ring surrounding the central vertex v. The k+1th ring is the difference between N(k-ring(v)) and k-disk(v).

[0044] According to the order defined in the spiral sequence S(v,n), the spiral graph convolutional network connects the vertex features of the (l-1) layer and passes them to the l layer. The propagation method is as follows:

[0045]

[0046] In the above formula, Represents the lth layer of the spiral graph convolutional network; represents the j-th vertex of the l-1-th layer of the spiral graph convolutional network; σ represents the nonlinear activation function, and || represents the connection operation.

[0047] The information transfer mechanism of the spiral graph convolutional network SpiralConv(v) is as follows:

[0048] SpiralConv(v)=σ(Wf(S(v,n))+b)

[0049] In the above formula, f(S(v,n)) represents the vertex features spliced ​​in the order of the spiral sequence, W represents the weight matrix shared by all vertices; b represents the bias value shared by all vertices.

[0050] As a further improvement of the present invention, in step S3, the local models of the body, hands, and face are generated by optimizing the corresponding parameterized network models and spiral graph convolutional networks based on the input feature vectors. Each parameterized network model and spiral graph convolutional network needs to be trained before application; the loss function during the training process is set as follows:

[0051] L = loss shape +loss 3d +loss 2d +λ edge loss edge +λ norm loss norm

[0052] In the above formula, loss shape Represents the point-to-point error between the true value model and the predicted model; loss 3d Represents the error between the true value of the 3D joint point and the predicted 3D joint point; loss 2d Represents the error between the true value of the 2D joint point and the predicted 2D joint point; and loss norm Respectively represent the edge length consistency loss and normal vector consistency loss between the true value and the triangle face in the prediction model, λ edge and λ norm Represent the loss term coefficients of edge length consistency loss and normal vector consistency loss respectively. The loss term coefficients during training are as follows: edge =20,λ norm =0.1.

[0053] In the present invention, loss shape The expression is as follows:

[0054] loss shape =||VV * ||1

[0055] In the above formula, V * Represents the point set in the true value model; V represents the point set in the prediction model.

[0056] loss 3d The expression is as follows:

[0057]

[0058] In the above formula, J represents the number of joint points in the local model; MSE() is the root mean square error function; X jThe 3D joint points are extracted from the predicted model using the joint point regression matrix by multiplying the model grid point matrix by the joint point regression matrix; are the 3D joint points of the ground-truth model, which are also extracted from the ground-truth model using the joint point matrix.

[0059] loss 2d The expression is as follows:

[0060]

[0061] In the above formula, v j Is a binary variable with a value of 0 or 1, which is used to indicate the visibility of each 2D joint point; x j Indicates projecting the predicted 3D joint points onto the 2D plane to obtain 2D joint points, x j =s(Π(X j )+t), s and t come from the camera parameters predicted by the network, s represents the zoom scale, t represents the translation vector, and Π represents that the projection method used is orthogonal projection; is the true value of the 2D joint point.

[0062] loss edge The expression is as follows:

[0063]

[0064] In the above formula, f represents the triangular face in the true value model or prediction model, m i and m jj They represent the i-th and j-th vertices in f respectively, and * is used to indicate that the value is true.

[0065] loss norm The expression is as follows:

[0066]

[0067] In the above formula, is the unit normal vector of the triangle face in the true model.

[0068] The present invention also includes a full-body human model reconstruction system based on a spiral graph convolutional network. This full-body human model reconstruction system utilizes the aforementioned full-body human model reconstruction method based on a spiral graph convolutional network. Based on a two-dimensional input image of a person, the system reconstructs a three-dimensional full-body model of the corresponding person, including facial expressions and gestures. This full-body human model reconstruction system includes an image preprocessing module, a feature extraction module, a spiral graph convolutional network model, a body reconstruction module, a hand reconstruction module, a face reconstruction module, and a model integration module.

[0069] The image preprocessing module uses the corresponding object detection algorithm to identify the corresponding body, hand, and face regions from the input person image. It then crops the identified target regions from the original image and resizes the cropped local images to a uniform size of 224*224, resulting in the corresponding body, hand, and face images.

[0070] The feature extraction module is obtained by training the improved ResNet50 convolutional neural network. The improvements to the ResNet50 convolutional neural network include removing the fully connected layer and using the average pooling layer output as the final output. The feature extraction module is used to extract features from the body image, hand image, and face image output by the image preprocessing module, and then obtain the corresponding 2048-dimensional feature vector F b 、F h 、F f .

[0071] The spiral graph convolutional network model is used to generate corresponding spiral sequences based on vertex features rich in image feature information in each human body local model, and predict the positions of each vertex used to generate the optimized human body local model based on the spiral sequence.

[0072] The body reconstruction module is used to first generate a reference topology of the body model by parameterizing the human mesh model through SMPL-X. Then the feature vector F b The vertex features containing image feature information are embedded into the vertices of the reference topology of the body model. These vertex features are then fed into a spiral graph convolutional network, which transfers information between vertices using a spiral convolutional approach to predict the position of each vertex. Finally, the predicted vertex positions are fed into the SMPL-X parameterized human mesh model to generate an optimized body model.

[0073] The hand reconstruction module is used to generate a reference topology of the hand model by parameterizing the hand mesh model through MANO. Then the feature vector F of the hand image is h The vertex features containing image feature information are embedded into the vertices of the reference topology of the hand model. The vertex features are input into the spiral graph convolutional network, which transmits information between vertices in a spiral convolution manner to predict the position of each vertex. Finally, the predicted vertex positions are input into the MANO parameterized hand mesh model to generate an optimized hand model.

[0074] The face reconstruction module is used to first generate a reference topology of the face model by parameterizing the face mesh model with FLAME, and then transform the feature vector F fThe vertex features containing image feature information are embedded into the vertices of the reference topology of the facial model. These vertex features are then fed into the spiral graph convolutional network, which transfers information between vertices using a spiral convolutional approach to predict the position of each vertex. Finally, the predicted vertex positions are fed into the FLAME parameterized facial mesh model to generate an optimized facial model.

[0075] The model integration module is used to integrate the reconstructed body model, hand model, and face model according to the vertex sequence defined in the SMPL-X parameterized human mesh model to obtain the required three-dimensional full-body model including facial expressions and gestures.

[0076] The present invention also includes a device for reconstructing a full-body human model based on a spiral graph convolutional network, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements the aforementioned steps of the method for reconstructing a full-body human model based on a spiral graph convolutional network, thereby reconstructing a full-body three-dimensional model of the corresponding person, including facial expressions and gestures, based on a two-dimensional image of the person being input.

[0077] The technical solution provided by the present invention has the following beneficial effects:

[0078] (1) This invention creatively applies spiral graph convolution processing to the prediction of model parameters, overcoming the problem that existing graph convolutional neural networks, which use spectral convolution on graph data structures, result in uneven and irregular reconstructed human body surfaces and slow reconstruction speeds. This approach is easier to train than graph convolutional neural networks using conventional convolution operators, and the resulting human body models are smoother and more accurate.

[0079] (2) The present invention uses a spiral graph convolutional neural network for model parameter prediction and design, which can directly predict the vertex coordinate positions in the human body mesh model, avoiding the need to predict complex three-dimensional posture rotation parameters. This also reduces the complexity of the final model integration. This facilitates more accurate restoration of human posture, improves the processing speed and real-time performance of the solution, and reduces the limitations of the solution on the data processing capabilities of hardware devices.

[0080] (3) This invention fills the gap in the research of graph convolutional neural networks in reconstructing a full-body expressive human body model. Current model-free methods are all based on SMPL and do not include the recovery of gestures and facial expressions. This invention is the first to fully apply graph convolutional neural networks to full-body 3D human mesh reconstruction. BRIEF DESCRIPTION OF THE DRAWINGS

[0081] Figure 1 This is a flowchart of the steps of a method for reconstructing a full-body human body model based on a spiral graph convolutional network provided in Example 1 of the present invention.

[0082] Figure 2 The flowchart of the processing method for extracting body images, hand images and face images from two-dimensional character images.

[0083] Figure 3 This is the model architecture diagram of the improved feature extraction network.

[0084] Figure 4 Schematic diagram of a case in which a corresponding 3D full-body model is generated based on a 2D character image in a test experiment.

[0085] Figure 5 Schematic diagram of the 3D human body reconstruction results of multiple different sample images.

[0086] Figure 6 This is a system architecture diagram of a full-body human model reconstruction system based on a spiral graph convolutional network provided in Example 2 of the present invention. DETAILED DESCRIPTION

[0087] In order to make the purpose, technical solutions and advantages of the present invention more clearly understood, the present invention will be further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0088] Example 1

[0089] This embodiment provides a method for reconstructing a full-body human model based on a spiral graph convolutional network, which is used to reconstruct a three-dimensional full-body model of the corresponding person including facial expressions and gestures based on a two-dimensional image of the person being input. Figure 1 As shown, the whole body model reconstruction method includes the following steps:

[0090] S1: Preprocess the input person image to extract the partial images corresponding to the human body, hand joints, and face respectively; and adjust the size of the obtained body image, hand image, and face image to 224*224.

[0091] like Figure 2 As shown, the process of extracting local images corresponding to different parts based on the character image in this embodiment is as follows:

[0092] S11: performing target detection on the input person image using a human body detection algorithm, a joint detection algorithm, or a face detection algorithm.

[0093] S12: Sort the pixels of the identified target area by coordinate value to obtain the maximum value x of the horizontal coordinate max and the minimum value x min , and the maximum value of the ordinate y max and the minimum value ymin ; Then determine the center coordinates of the detection box of each target area and the cropping scale:

[0094]

[0095] S13: According to the center coordinates of the detection frame corresponding to different parts of the person and the cropping scale, the cropped local images I are obtained through the affine transformation function. b for:

[0096] I b =(I; center, scale),

[0097] Where I represents the original input person image.

[0098] S14: setting the size of each cropped local image to 224*224.

[0099] S2: Remove the fully connected layer in the ResNet50 convolutional neural network, use the average pooling layer output as the final output, and use it as the required feature extraction network. The network architecture of the improved feature extraction network in this embodiment is as follows: Figure 3 As shown in the figure, the pre-trained feature extraction network is used to extract features of the body image, hand image and face image respectively, and the corresponding 2048-dimensional feature vector F is obtained. b 、F h 、F f .

[0100] S3: Based on the reference topology of each part of the human body and the extracted feature vectors, a spiral graph convolutional network is used to process the graph structure and predict the position of each vertex to reconstruct the corresponding local model. The specific process is as follows:

[0101] S31: The corresponding parameterized grid model is used as the reference topology structure of the spiral graph convolutional network.

[0102] S32: embedding the feature vectors extracted from each local image into each vertex of the parameterized network model to obtain vertex features containing image feature information.

[0103] S33: Input the vertex features into the spiral graph convolutional network, transfer information between vertices in the form of spiral convolution, and finally output the predicted position of each vertex.

[0104] S34: Based on the vertex positions predicted by the spiral graph convolutional network, the corresponding 3D local models are generated in the parameterized mesh model; including the body model, hand model and face model.

[0105] S4: Integrate the reconstructed body model, hand model, and face model according to the vertex sequence defined in the SMPL-X parameterized human mesh model to obtain the required three-dimensional full-body model including facial expressions and gestures.

[0106] In step S3 of this embodiment, the body model, hand model, and face model are three synchronously processed in parallel. The reconstruction process of the body model is as follows:

[0107] S31A: The SMPL-X parameterized human mesh model is used as the reference topology for the spiral graph convolutional network.

[0108] S32A: The feature vector F of the body image b Embedded into the vertices of the SMPL-X human body mesh model to obtain vertex features containing image feature information.

[0109] S33A: Input vertex features into the spiral graph convolutional network, transfer information between vertices in the form of spiral convolution, and finally output the predicted position of each vertex.

[0110] S34A: Generate a corresponding body model in the SMPL-X parameterized human mesh model based on the vertex positions predicted by the spiral graph convolutional network.

[0111] Accordingly, the reconstruction process of the hand model is as follows:

[0112] S31B: The MANO parameterized hand mesh model is used as the reference topology for the spiral graph convolutional network.

[0113] S32B: The feature vector F of the facial image h Embedded into the vertices of the MANO parameterized hand mesh model, vertex features containing image feature information are obtained.

[0114] S33B: Input vertex features into the spiral graph convolutional network, transfer information between vertices in the form of spiral convolution, and finally output the predicted position of each vertex.

[0115] S34B: Generate a corresponding hand model in the MANO parameterized hand mesh model based on the vertex positions predicted by the spiral graph convolutional network.

[0116] Accordingly, the reconstruction process of the face model is as follows:

[0117] S31C: The FLAME parameterized face mesh model is used as the reference topology for the spiral graph convolutional network.

[0118] S32C: The feature vector F of the facial image fEmbedded into the vertices of the FLAME parameterized face mesh model to obtain vertex features containing image feature information.

[0119] S33C: Input vertex features into the spiral graph convolutional network, transfer information between vertices in the form of spiral convolution, and finally output the predicted position of each vertex.

[0120] S34C: Generates a corresponding face model in the FLAME parameterized face mesh model based on the vertex positions predicted by the spiral graph convolutional network.

[0121] Existing model reconstruction methods based on graph convolutional neural networks all use spectral convolution on graph data structures. Graph convolutional networks are a method for extracting features from graph structures and can effectively process data in non-Euclidean spaces. Traditional graph convolutional networks aggregate features of adjacent vertices using an adjacency matrix defined on the graph structure. This is essentially equivalent to performing a full connection on the features of each vertex in the graph and then applying a neighborhood averaging operation. This often results in uneven and irregular reconstructed human body surfaces and slows down the reconstruction of human body models.

[0122] In the solution provided in this embodiment, a spiral graph convolutional network replaces the conventional graph convolutional neural network. The spiral graph convolutional network does not use the graph adjacency matrix to perform feature aggregation, but instead explicitly defines the order in which adjacent vertex features are aggregated. This order is called a spiral sequence.

[0123] In the generation stage of the body model, hand model and face model in step S3, the spiral graph convolutional network can construct the aggregation order of adjacent vertex features in a spiral manner based on the central vertex in the given graph structure to obtain the corresponding spiral sequence S(v,n).

[0124] In order to define a spiral sequence, we first need to introduce several concepts. Given a central vertex v in a graph structure, the kth ring and kth disk surrounding this central vertex v are defined as follows:

[0125] 0-ing(v)={v}#(1)

[0126] k-disk(v)=∪ i＝0,,…, i-ring(v)#(2)

[0127]

[0128] In formula #(3) Represents the adjacent vertices of all vertices in the vertex set V.

[0129] Assuming that the length of a spiral sequence is n, the spiral sequence S(v,n) is defined as follows:

[0130]

[0131] This formula indicates that a spiral sequence is an ordered set of n vertices v in k rings.

[0132] In a spiral graph convolutional network, like the adjacency matrix of a traditional graph convolutional network, the spiral sequence is only calculated once at network initialization and remains unchanged thereafter. Thus, in a spiral graph convolutional network, the way features are propagated between layers is defined as follows:

[0133]

[0134] This formula defines the feature propagation method between layer l and layer l-1 in the spiral graph convolutional network. σ represents the nonlinear activation function, || represents the connection operation, and this formula means that the vertex features of the (l-1)th layer are connected and passed to the lth layer in the order defined in the spiral sequence S(v,n).

[0135] This formula defines the information transfer mechanism of spiral graph convolution from a theoretical perspective. In actual calculations, the formula can be transformed into:

[0136] SpiralConv(v)=σ(Wf(S(v,n))+b)#(6)

[0137] In the above formula, f(S(v,n)) represents the vertex features spliced ​​in the order of the spiral sequence, and W and b are the weight matrix and bias value shared by all vertices.

[0138] In this embodiment, the local models of the body, hands, and face are generated by optimizing the corresponding parameterized network models and spiral graph convolutional networks based on the input feature vectors. Each parameterized network model and spiral graph convolutional network must be trained separately before application; the loss function during training is set as follows:

[0139] L = loss shape +loss 3d +loss 2d +λ edge loss edge +λ norm loss norm

[0140] In the above formula, loss shape Represents the point-to-point error between the true value model and the predicted model; loss 3d Represents the error between the true value of the 3D joint point and the predicted 3D joint point; loss 2d Represents the error between the true value of the 2D joint point and the predicted 2D joint point; and loss normRespectively represent the edge length consistency loss and normal vector consistency loss between the true value and the triangle face in the prediction model, λ edge and λ norm Represent the loss term coefficients of edge length consistency loss and normal vector consistency loss respectively. The loss term coefficients during training are as follows: edge =20,λ norm =0.1.

[0141] In the present invention, loss shape The expression is as follows:

[0142] loss shape =||VV * ||1

[0143] In the above formula, V * Represents the point set in the true value model; V represents the point set in the prediction model.

[0144] loss 3d The expression is as follows:

[0145]

[0146] In the above formula, J represents the number of joint points in the local model; MSE() is the root mean square error function; X j The 3D joint points are extracted from the predicted model using the joint point regression matrix by multiplying the model grid point matrix by the joint point regression matrix; are the 3D joint points of the ground-truth model, which are also extracted from the ground-truth model using the joint point matrix.

[0147] loss 2d The expression is as follows:

[0148]

[0149] In the above formula, v j Is a binary variable with a value of 0 or 1, which is used to indicate the visibility of each 2D joint point; x j Indicates projecting the predicted 3D joint points onto the 2D plane to obtain 2D joint points, x j =s(Π(X j )+t), s and t come from the camera parameters predicted by the network, s represents the zoom scale, t represents the translation vector, and Π represents the orthogonal projection method; is the true value of the 2D joint point.

[0150] loss edge The expression is as follows:

[0151]

[0152] In the above formula, f represents the triangular face in the true value model or prediction model, m i and m jj They represent the i-th and j-th vertices in f respectively, and * is used to indicate that the value is true.

[0153] loss norm The expression is as follows:

[0154]

[0155] In the above formula, is the unit normal vector of the triangle face in the true model.

[0156] In the solution provided in this embodiment, if the target person in the 2D image is symmetrical, only one hand image needs to be preprocessed to generate a hand model. This hand model is then integrated into the other side of the 3D full-body model during the model integration phase. If the target person in the 2D image is asymmetrical, it is necessary to extract the left and right hand images separately, generate corresponding left and right hand models, and then perform model integration.

[0157] Performance Testing

[0158] In order to verify the effectiveness of the whole-body model reconstruction method based on spiral graph convolutional network provided by this embodiment, this embodiment designed the following simulation experiment to test the method of this embodiment.

[0159] Specifically, in the experimental simulation stage, the deep learning framework Pytorch was selected to complete the specific implementation of the program. The hardware environment for the program running was: Intel Core i5-10400f@2.90GHz twelve-thread CPU, 16G DDR4 memory, Ubuntu 18.04 operating system, Pycharm development tool, and NVIDIA RTX 3060 GPU for training the network.

[0160] The training datasets used in the test experiments are publicly available datasets, including Human3.6M, MSCOCO, MPII, FreiHAND, HO3D-V2, and VGGFace2. The body reconstruction network is trained using Human3.6M, MSCOCO, and MPII. Since these three datasets use SMPL parameterized models, we use the NeuralAnnot method to obtain the ground-truth 3D human mesh in the SMPL-X model format. The hand reconstruction network is trained using the FreiHAND and HO3D-V2 datasets, and VGGFace2 is used to train the face reconstruction network.

[0161] For training, this experiment used the Adam optimizer with a learning rate of 0.0001, no learning rate decay, and a batch size of 16. For the body reconstruction network, we trained for a total of 35 epochs. Edge consistency loss and normal consistency loss were not used for supervision in the early stages of training, as they can cause training to fall into local optima. Instead, these two losses were added midway through training. The hand reconstruction network was trained for 60 epochs, and the face reconstruction network for 80 epochs. Data augmentation strategies were employed during training, including image rotation, horizontal flipping, noise addition, and scaling.

[0162] Figure 4 During the test, the present embodiment is used to generate a three-dimensional model of the body reconstruction result, hand reconstruction result, face reconstruction result, and the corresponding whole-body reconstruction result based on the corresponding two-dimensional character image.

[0163] In addition, using the same method, this embodiment also uses multiple different sample images to reconstruct the corresponding three-dimensional human body model. Figure 5 As shown in the figure, the first row shows the modeling results for full-body images, the second row shows the modeling results for hand images, and the third row shows the modeling results for facial images. As can be seen from the results in the figure, the solution of this embodiment produces excellent modeling results for both full-body and local images, and the generated 3D models are very detailed.

[0164] Example 2

[0165] Based on Example 1, this example further improves a full-body human model reconstruction system based on a spiral graph convolutional network. The full-body human model reconstruction system adopts the full-body human model reconstruction method based on a spiral graph convolutional network as in Example 1, and reconstructs a three-dimensional full-body model of the corresponding person including facial expressions and gestures based on the input two-dimensional image of the person.

[0166] like Figure 6 As shown, the whole-body model reconstruction system provided in this embodiment includes: an image preprocessing module, a feature extraction module, a spiral graph convolutional network model, a body reconstruction module, a hand reconstruction module, a face reconstruction module, and a model integration module.

[0167] The image preprocessing module uses the corresponding object detection algorithm to identify the corresponding body, hand, and face regions from the input person image. It then crops the identified target regions from the original image and resizes the cropped local images to a uniform size of 224*224, resulting in the corresponding body, hand, and face images.

[0168] The feature extraction module is obtained by training the improved ResNet50 convolutional neural network. The improvements to the ResNet50 convolutional neural network include removing the fully connected layer and using the average pooling layer output as the final output. The feature extraction module is used to extract features from the body image, hand image, and face image output by the image preprocessing module, and then obtain the corresponding 2048-dimensional feature vector F b 、F h 、F f .

[0169] The spiral graph convolutional network model is used to generate corresponding spiral sequences based on vertex features rich in image feature information in each human body local model, and predict the positions of each vertex used to generate the optimized human body local model based on the spiral sequence.

[0170] The body reconstruction module is used to first generate a reference topology of the body model by parameterizing the human mesh model through SMPL-X. Then the feature vector F b The vertex features containing image feature information are embedded into the vertices of the reference topology of the body model. These vertex features are then fed into a spiral graph convolutional network, which transfers information between vertices using a spiral convolutional approach to predict the position of each vertex. Finally, the predicted vertex positions are fed into the SMPL-X parameterized human mesh model to generate an optimized body model.

[0171] The hand reconstruction module is used to generate a reference topology of the hand model by parameterizing the hand mesh model through MANO. Then the feature vector F of the hand image is h The vertex features containing image feature information are embedded into the vertices of the reference topology of the hand model. The vertex features are input into the spiral graph convolutional network, which transmits information between vertices in a spiral convolution manner to predict the position of each vertex. Finally, the predicted vertex positions are input into the MANO parameterized hand mesh model to generate an optimized hand model.

[0172] The face reconstruction module is used to first generate a reference topology of the face model by parameterizing the face mesh model with FLAME, and then transform the feature vector F f The vertex features containing image feature information are embedded into the vertices of the reference topology of the facial model. These vertex features are then fed into the spiral graph convolutional network, which transfers information between vertices using a spiral convolutional approach to predict the position of each vertex. Finally, the predicted vertex positions are fed into the FLAME parameterized facial mesh model to generate an optimized facial model.

[0173] The model integration module is used to integrate the reconstructed body model, hand model, and face model according to the vertex sequence defined in the SMPL-X parameterized human mesh model to obtain the required three-dimensional full-body model including facial expressions and gestures.

[0174] Example 3

[0175] Based on Examples 1 and 2, this embodiment further provides a device for reconstructing a full-body human model based on a spiral graph convolutional network. This device is a physical data processing device capable of performing the method described in Example 1. The device includes a memory, a processor, and a computer program stored in the memory and executable by the processor. When the processor executes the computer program, it implements the steps of the aforementioned method for reconstructing a full-body human model based on a spiral graph convolutional network, thereby reconstructing a three-dimensional full-body model of the corresponding person, including facial expressions and gestures, based on a two-dimensional image of the input person.

[0176] The computer device may be a smartphone, tablet computer, laptop computer, desktop computer, rack server, blade server, tower server, or cabinet server (including a standalone server or a server cluster consisting of multiple servers) that can execute programs. The computer device of this embodiment includes at least, but is not limited to, a memory and a processor that can be interconnected via a system bus.

[0177] In this embodiment, the memory (i.e., readable storage medium) includes flash memory, hard disk, multimedia card, card-type memory (e.g., SD or DX memory), random access memory (RAM), static random access memory (SRAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), programmable read-only memory (PROM), magnetic storage, magnetic disk, optical disk, etc. In some embodiments, the memory may be an internal storage unit of a computer device, such as the hard disk or internal memory of the computer device. In other embodiments, the memory may also be an external storage device of the computer device, such as a plug-in hard disk equipped with the computer device, a smart media card (SMC), a secure digital (SD) card, a flash memory card, etc. Of course, the memory may also include both the internal storage unit of the computer device and its external storage device. In this embodiment, the memory is generally used to store the operating system and various application software installed on the computer device. In addition, the memory may also be used to temporarily store various types of data that have been output or are about to be output.

[0178] In some embodiments, the processor may be a central processing unit (CPU), a controller, a microcontroller, a microprocessor, or other data processing chip. The processor is generally used to control the overall operation of a computer device.

[0179] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A full-body human model reconstruction method based on a spiral graph convolutional network, which is used to reconstruct a three-dimensional full-body model of the corresponding person including facial expressions and gestures based on a two-dimensional image of the person input, characterized in that: The whole body model reconstruction method comprises the following steps: S1: Preprocess the input person image to extract the partial images corresponding to the whole body, hand joints and face, and resize the obtained body image, hand image and face image to 224*224; S2: Remove the fully connected layer in the ResNet50 convolutional neural network and use the average pooling layer output as the final output, which is used as the required feature extraction network. The pre-trained feature extraction network is used to extract features from the body image, hand image, and face image, respectively, to obtain the corresponding 2048-dimensional feature vector F. b 、F h 、F f ; S3: Based on the reference topological structure of each part of the human body and the extracted feature vectors, a spiral graph convolutional network is used to process the graph structure and predict the position of each vertex to reconstruct the corresponding local model. The specific process is as follows: S31: The corresponding parameterized grid model is used as the reference topology of the spiral graph convolutional network; S32: embedding the feature vectors extracted from each local image into each vertex of the parameterized network model to obtain vertex features containing image feature information; S33: Input the vertex features into the spiral graph convolutional network, transfer information between vertices in the form of spiral convolution, and finally output the predicted position of each vertex; S34: Based on the vertex positions predicted by the spiral graph convolutional network, generating a corresponding three-dimensional local model in the parameterized mesh model; the three-dimensional local model includes a body model, a hand model, and a face model; S4: Integrate the reconstructed body model, hand model, and face model according to the vertex sequence defined in the SMPL-X parameterized human mesh model to obtain the required three-dimensional full-body model including facial expressions and gestures.

2. The method for reconstructing a whole-body human body model based on a spiral graph convolutional network according to claim 1, wherein: In step S1, the process of extracting local images corresponding to different parts based on the person image is as follows: S11: performing target detection on the input person image using a human body detection algorithm, a joint detection algorithm, or a face detection algorithm; S12: Sort the pixels of the identified target area by coordinate value to obtain the maximum value x of the horizontal coordinate max and the minimum value x min , and the maximum value of the ordinate y max and the minimum value y min ; Then determine the center coordinates of the detection box of each target area and the cropping scale: S13: According to the center coordinates of the detection frame corresponding to different parts of the person and the cropping scale, the cropped local images I are obtained through the affine transformation function. b for: I b =T(I;center,scale), Where I represents the original input character image; S14: setting the size of each cropped local image to 224*224.

3. The method for reconstructing a whole-body human body model based on a spiral graph convolutional network according to claim 1, wherein: In step S3, the body model is reconstructed as follows: S31A: The SMPL-X parameterized human mesh model is used as the reference topology for the spiral graph convolutional network. S32A: The feature vector F of the body image b Embed it into the vertices of the SMPL-X human body mesh model to obtain vertex features containing image feature information; S33A: Input vertex features into the spiral graph convolutional network, transfer information between vertices in a spiral convolution manner, and finally output the predicted position of each vertex; S34A: Generate a corresponding body model in the SMPL-X parameterized human mesh model based on the vertex positions predicted by the spiral graph convolutional network.

4. The method for reconstructing a whole-body human body model based on a spiral graph convolutional network according to claim 1, wherein: In step S3, the reconstruction process of the hand model is as follows: S31B: The MANO parameterized hand mesh model is used as the reference topology for the spiral graph convolutional network. S32B: The feature vector F of the facial image h Embedded into the vertices of the MANO parameterized hand mesh model to obtain vertex features containing image feature information; S33B: Input vertex features into the spiral graph convolutional network, transfer information between vertices in a spiral convolution manner, and finally output the predicted position of each vertex; S34B: Generate a corresponding hand model in the MANO parameterized hand mesh model based on the vertex positions predicted by the spiral graph convolutional network.

5. The method for reconstructing a whole-body human body model based on a spiral graph convolutional network according to claim 1, wherein: In step S3, the reconstruction process of the face model is as follows: S31C: Using the FLAME parameterized face mesh model as the reference topology for the spiral graph convolutional network; S32C: The feature vector F of the facial image f Embedded into the vertices of the FLAME parameterized face mesh model to obtain vertex features containing image feature information; S33C: Input vertex features into the spiral graph convolutional network, transfer information between vertices in the form of spiral convolution, and finally output the predicted position of each vertex; S34C: Generates a corresponding face model in the FLAME parameterized face mesh model based on the vertex positions predicted by the spiral graph convolutional network.

6. The method for reconstructing a whole-body human body model based on a spiral graph convolutional network according to claim 1, wherein: In the spiral graph convolutional network of step S3, based on the central vertex in the given graph structure, the aggregation order of adjacent vertex features is constructed in a spiral manner to obtain the corresponding spiral sequence S(v,n): In the above formula, the spiral sequence S(v,n) is an ordered set of n vertices v in k rings; 0-ring(v) = {v} represents the zeroth ring around the central vertex v in a given graph structure, and the zeroth ring is the central vertex v; k-disk(v) = ∪ i＝0，1，...，k i-ring(v) represents the k-th disk surface around the central vertex v. The k-th disk consists of the union of the 0th ring, the 1st ring, and the k-th ring. N(k-ring(v)) represents the set of adjacent vertices of all vertices in the k-th ring. (k+1)-ring(v) represents the k+1th ring around the central vertex v. The k+1th ring is the difference between N(k-ring(v)) and k-disk(v). According to the order defined in the spiral sequence S(v,n), the spiral graph convolutional network connects the vertex features of the (l-1) layer and passes them to the l layer. The propagation method is as follows: In the above formula, Represents the lth layer of the spiral graph convolutional network; represents the j-th vertex of the l-1-th layer of the spiral graph convolutional network; σ represents the nonlinear activation function, ‖ represents the connection operation; The information transfer mechanism of the spiral graph convolutional network SpiralConv(v) is as follows: SpiralConv(v)=σ(W·f(S(v,n))+b) In the above formula, f(S(v,n)) represents the vertex features spliced ​​in the order of the spiral sequence, W represents the weight matrix shared by all vertices; b represents the bias value shared by all vertices.

7. The method for reconstructing a whole-body human body model based on a spiral graph convolutional network according to claim 1, wherein: In step S3, the local models of the body, hands, and face are generated by optimizing the corresponding parameterized network models and spiral graph convolutional networks based on the input feature vectors. Each parameterized network model and spiral graph convolutional network needs to be trained before application. The loss function during the training process is set as follows: L=loss shape +loss 3d +loss 2d +λ edge loss edge +λ norm loss norm In the above formula, loss shape Represents the point-to-point error between the true value model and the predicted model; loss 3d Represents the error between the true value of the 3D joint point and the predicted 3D joint point; loss 2d Represents the error between the true value of the 2D joint point and the predicted 2D joint point; and loss norm Respectively represent the edge length consistency loss and normal vector consistency loss between the true value and the triangle face in the prediction model, λ edge and λ norm Represent the loss term coefficients of edge length consistency loss and normal vector consistency loss respectively. The loss term coefficients during training are as follows: edge =20,λ norm =0.

1.

8. The method for reconstructing a whole-body human body model based on a spiral graph convolutional network according to claim 7, wherein: loss shape The expression is as follows: loss shape =‖V-V * ‖1 In the above formula, V * Represents the point set in the true value model; V represents the point set in the prediction model; loss 3d The expression is as follows: In the above formula, J represents the number of joint points in the local model; MSE() is the root mean square error function; X j The 3D joint points are extracted from the predicted model using the joint point regression matrix by multiplying the model grid point matrix by the joint point regression matrix; is the 3D joint point of the ground-truth model, which is also extracted from the ground-truth model using the joint point matrix; loss 2d The expression is as follows: In the above formula, v j Is a binary variable with a value of 0 or 1, which is used to indicate the visibility of each 2D joint point; x j Indicates projecting the predicted 3D joint points onto the 2D plane to obtain 2D joint points, x j =s(Π(X j )+t), s and t come from the camera parameters predicted by the network, s represents the zoom scale, t represents the translation vector, and Π represents the orthogonal projection method; is the true value of the 2D joint point; loss edge The expression is as follows: In the above formula, f represents the triangular face in the true value model or prediction model, m i and m j Represent the i-th and j-th vertices in f respectively, and * is used to indicate that this value is true; loss norm The expression is as follows: In the above formula, is the unit normal vector of the triangle face in the true model.

9. A full-body human model reconstruction system based on spiral graph convolutional networks, characterized by: The method for reconstructing a full-body human model based on a spiral graph convolutional network as claimed in any one of claims 1 to 8 is used to reconstruct a three-dimensional full-body model of the corresponding person including facial expressions and gestures according to a two-dimensional image of the input person; The whole body model reconstruction system comprises: The image preprocessing module is used to use the corresponding target detection algorithm to identify the corresponding body, hand and face areas from the input person image, then crop the identified target areas in the original image and resize the cropped local images to a uniform size of 224*224 to obtain the corresponding body images, hand images and face images; The feature extraction module is obtained by training the improved ResNet50 convolutional neural network. The improvement includes removing the fully connected layer and using the average pooling layer output as the final output. The feature extraction module is used to extract features from the body image, hand image and face image output by the image preprocessing module, and then obtain the corresponding 2048-dimensional feature vector F b 、F h 、F f ; A spiral graph convolutional network model is used to generate a corresponding spiral sequence based on vertex features rich in image feature information in each human body local model, and predict the position of each vertex used to generate an optimized human body local model based on the spiral sequence; The body reconstruction module is used to first generate a reference topology of the body model by parameterizing the human mesh model through SMPL-X, and then convert the feature vector F b The method embeds the vertices of the reference topological structure of the body model to obtain vertex features containing image feature information; the vertex features are input into the spiral graph convolutional network, and information is transferred between vertices in a spiral convolution manner to obtain the predicted position of each vertex; finally, the predicted vertex positions are input into the SMPL-X parameterized human body mesh model to generate an optimized body model; The hand reconstruction module is used to first generate a reference topology of the hand model by parameterizing the hand mesh model through MANO, and then transform the feature vector F h The hand model is embedded into the vertices of the reference topological structure to obtain vertex features containing image feature information; the vertex features are input into the spiral graph convolution network, and information is transferred between vertices in a spiral convolution manner to obtain the predicted position of each vertex; finally, the predicted vertex position is input into the MANO parameterized hand mesh model to generate an optimized hand model; The face reconstruction module is used to first generate a reference topology of the face model by parameterizing the face mesh model with FLAME, and then convert the feature vector F f Embedding into the vertices of the reference topological structure of the facial model to obtain vertex features containing image feature information; inputting the vertex features into the spiral graph convolutional network, transferring information between vertices in a spiral convolution manner, and thereby predicting the position of each vertex; finally, inputting the predicted vertex positions into the FLAME parameterized facial mesh model to generate an optimized facial model; and The model integration module is used to integrate the reconstructed body model, hand model, and face model according to the vertex sequence defined in the SMPL-X parameterized human mesh model to obtain the required three-dimensional full-body model including facial expressions and gestures.

10. A device for reconstructing a full-body human body model based on a spiral graph convolutional network, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, it implements the steps of the full-body human body model reconstruction method based on the spiral graph convolutional network as described in any one of claims 1 to 8, and then reconstructs a three-dimensional full-body model of the corresponding person including facial expressions and gestures based on the input two-dimensional image of the person.

Images