A method for three-dimensional reconstruction of human body based on image

Abstract

The application discloses a human body three-dimensional reconstruction method based on images, which comprises a posture fusion three-dimensional human body reconstruction network and a graph neural radiation field three-dimensional human body reconstruction network. After the posture fusion three-dimensional human body reconstruction network and the graph neural radiation field three-dimensional human body reconstruction network respectively process and reconstruct input images, the posture fusion three-dimensional human body reconstruction network and the graph neural radiation field three-dimensional human body reconstruction network are fused to realize final three-dimensional reconstruction. The posture fusion three-dimensional human body reconstruction network comprises a surface normal map generation module, a surface normal map feature coding module, a human body posture feature coding module, a feature decoding reconstruction module, a back image generation module and a texture mapping module. The graph neural radiation field three-dimensional human body reconstruction network comprises a posture optimization module, a human body image coding module, a feature decoding module and an integration module. The application improves the reconstruction quality of a human body three-dimensional model and a human body new view image.

Description

Technical Field

[0001] This invention relates to the field of three-dimensional reconstruction technology, and in particular to a method for three-dimensional human reconstruction using a posture-fusion three-dimensional human reconstruction network and a graph neural radiation field three-dimensional human reconstruction network. Background Technology

[0002] Image-based 3D human reconstruction refers to the process of reconstructing a 3D human model from one or more 2D human images. It is a research area with high value in computer vision.

[0003] Deep learning-based 3D human body reconstruction methods leverage massive amounts of data to build prior knowledge, transforming the 3D reconstruction process into encoding and decoding, directly reconstructing the human body from 2D images. Early 3D human body reconstruction methods reconstructed parametric models of the human body, but these models could only represent the 3D pose of the body, failing to represent detailed information such as clothing. To achieve better reconstruction performance and generalization ability, implicit function-based 3D human body reconstruction methods use stacked hourglass networks to encode images, decode implicit functions using multilayer perceptrons, predict the distribution of 3D points inside and outside the human body model, construct implicit functions in the form of point labels, and use extraction algorithms to extract mesh information from the implicit representation to reconstruct the human body model.

[0004] 3D reconstruction methods based on neural radiation fields aggregate image features from multiple perspectives, obtain color and volume density values ​​through a decoder, and render the image in a volumetric manner. These methods are trained on multiple scenes or specific target classes to perform 3D reconstruction from images from one or more perspectives. Some studies combine human parametric models with neural radiation fields to establish neural radiation fields from human videos and render images of the human body from other perspectives. However, global feature encoders are sensitive to global pose, and methods using only global feature encoders perform poorly in reconstructing complex human poses. Using multilayer perceptrons for feature decoding ignores the relationships between 3D points, resulting in poor reconstruction of detailed human body parts. Therefore, accurately extracting image features and human pose features and performing efficient feature decoding to improve generalization ability across different poses is key to improving the performance of 3D human body reconstruction. Summary of the Invention

[0005] The purpose of this invention is to address the problems of poor reconstruction performance and insufficient generalization ability in existing technologies by providing an image-based three-dimensional human body reconstruction method, which aims to efficiently extract features from human body images and obtain a high-quality three-dimensional human body model.

[0006] The technical solution adopted to achieve the purpose of this invention is:

[0007] The image-based 3D human reconstruction method involves two separate 3D human reconstruction networks: a pose fusion network and a graph neural radiation field network. The input images are processed and reconstructed individually, then fused to achieve the final 3D reconstruction. The pose fusion network includes a surface normal map generation module, a surface normal map feature encoding module, a human pose feature encoding module, a feature decoding and reconstruction module, a back view image generation module, and a texture mapping module. The graph neural radiation field network includes a pose optimization module, a human image encoding module, a feature decoding module, and an integration module.

[0008] The processing steps are as follows:

[0009] From human image X i Predicting the human parameter model M using posture estimation methods smp The vector map N of the frontal normal of the human body parameterized model is obtained by feeding it into a differentiable renderer. sf and the back normal map N of the human body parameterized model sb , with human body image X i The data are fed into the surface normal map generation module to obtain the frontal normal map N of the human body. cf and the normal diagram of the back of the human body N cb The surface normal map feature encoding module is used to extract the human body surface normal map features f. i ; The human body parameterization model M smp The data is fed into the human posture feature encoding module to extract local feature codes f. lo and volume feature encoding f v , and the characteristics of the human body surface normal map f i The values ​​are fed together into the feature decoding and reconstruction module to obtain the symbolic distance function value s. d And reconstruct a colorless, textureless 3D human body model Y s Output: Human body image X i and the normal diagram of the back of the human body N cb The images are fed together into the back image generation module to generate a human back image X. b Human body image X i Image of the back of the human body X b And a colorless and textureless 3D human body model Y s The data are fed together into the texture mapping module to reconstruct a 3D human body model Y with front and back color textures. c Output;

[0010] Human image X extracted by human image encoding module i Image features f r Image features f r With human body image X i Perform grid sampling interpolation and feature concatenation separately to obtain the image feature code f. ca; X-ray the human body image i The pixels are converted into three-dimensional point coordinates p in the neural radiation field, and positional encoding is added to obtain three-dimensional point features f with positional encoding. l and pose features f s The optimized attitude code f is obtained by sending it to the attitude optimization module. rs The orientation code f of the 3D point is calculated from the camera parameters. v , and image feature encoding f ca Posture characteristics f rs and three-dimensional point features f l The two data points are fed into the feature decoding module to obtain the color value c and the shape feature f. σ ; the symbolic distance function value s d and shape features f σ The data are fed into the integrated module to obtain the volume density value σ, and then rendered and output as a human body image Y corresponding to the preset viewpoint of the camera parameters. n .

[0011] This invention employs a surface normal map feature encoding module for feature extraction. Stacked network units are used to encode the human body surface normal map, increasing the network depth and fusing features from different stages. A channel attention layer is added to the residual encoding, and a Transformer network is used to capture long-distance dependencies, thereby obtaining the relationships between different parts of the human body surface normal map and improving the efficiency of feature extraction.

[0012] This invention employs a human posture feature encoding module to extract local feature codes and volume feature codes from a human parametric model, calculates the signed distance function of the human parametric model to obtain local feature codes, voxels the human parametric model, obtains volume feature codes from the human parametric model through three-dimensional residual encoding, and combines the local feature codes and volume feature codes of the human parametric model to reconstruct a human three-dimensional model with high precision and completeness.

[0013] This invention employs a feature decoding and reconstruction module for feature decoding. The module feeds the human body surface normal graph features, local feature codes, and volume feature codes into the feature decoder to obtain the signed distance function value of the human 3D model. A colorless, textureless human 3D model is then reconstructed using the moving cubes algorithm. The feature decoder consists of graph convolutional coding and one-dimensional convolutional coding. It utilizes the KNN algorithm to construct a graph structure of 3D points, obtaining the relationships between adjacent 3D points for more efficient feature decoding.

[0014] This invention employs a texture mapping module, uses mesh sampling interpolation to extract the color code of the human body model, rasterizes the mesh model, selects non-repeating values ​​to obtain the visibility of the human body model, uses the color code and visibility of the human body model to obtain the front and back color textures of the 3D model, and simultaneously uses the human body image and the back image of the human body to perform texture mapping, thereby improving the completeness of the reconstructed texture and obtaining a 3D human body model with front and back color textures.

[0015] This invention employs a pose optimization module, which uses the KNN algorithm to construct a pose feature map structure corresponding to high-dimensional pose features. The pose feature map structure is then fed into graph convolutional encoding to obtain high-dimensional pose optimization features, thereby improving the pose optimization effect.

[0016] This invention employs a human image encoding module, using stacked network units to encode human images, increasing the network depth and fusing features from different stages. It divides image features into multiple parts and uses the KNN algorithm to construct a graph structure of image features, obtaining the relationships between different parts of the human image, improving the efficiency of feature extraction, providing more accurate human image features for generating images from new perspectives, and improving the effect of generating images from new perspectives.

[0017] Among them, the human body image X i Predicting the human parameter model M using posture estimation methods smp The vector map N of the frontal normal of the human body parameterized model is obtained by feeding it into a differentiable renderer. sf and the back normal map N of the human body parameterized model sb The expression is as follows:

[0018]

[0019] Among them, G s (·) denotes the attitude estimation operation, R f (·),R b (·) indicates the operation of rendering the front and back normal maps of a parametric human body model.

[0020] The human body image X i Human body parametric model frontal normal image N sf and the human body parametric model back normal image N sb The vectors are fed into the surface normal map generation module to obtain the human body frontal normal map N. cf and the normal diagram of the back of the human body N cb The steps are as follows:

[0021] X i With the frontal normal map N of the human body parameterized model sfAfter feature concatenation, the data is fed into an image encoder to obtain the encoded features f representing the frontal normal map of the human body parameterized model. sf X-ray the human body image i With the frontal normal map N of the human body parameterized model sf After feature concatenation, the data is fed into an image encoder to obtain the encoded backface normal map features f of the human body parameterized model. sb Encoding features of the frontal normal map and the back normal map of the human body parametric model f sf ,f sb After being processed by the residual encoder, the images are upsampled to obtain the frontal normal image N of the human body. cf and the normal diagram of the back of the human body N cb ; indicates the following:

[0022]

[0023] Among them, f sf ,f sb E represents the frontal normal map encoding features and the back normal map encoding features of the human body parametric model, respectively. s (·) represents the image encoder operation. The image encoder consists of convolutional layers, instance normalization, and a ReLU activation function. F cat (·) indicates a feature concatenation operation, E res (·) denotes the residual encoder operation. The residual encoder consists of convolutional layers, channel attention layers, batch normalization, and ReLU activation functions. F up (·) indicates an upsampling operation, which consists of a transposed convolutional layer, batch normalization, and a ReLU activation function.

[0024] Among them, the surface normal map feature encoding module extracts the human body surface normal map feature f. i The steps are as follows:

[0025] Encode the frontal and back normal images of the human body separately to obtain the feature codes for the frontal and back normal images. Concatenate these feature codes and perform grid sampling interpolation to obtain the human surface normal image feature f. i :

[0026]

[0027] Among them, E c(·) represents the human surface normal map encoding operation of the surface normal map encoding module. The surface normal map encoding module consists of stacked network units, residual units, and convolutional layers. The stacked network units include max pooling layers, Transformer encoding layers, and upsampling layers. The Transformer encoding layers include multi-head attention layers, layer normalization, and multi-layer perceptron layers. The residual units include convolutional layers and ReLU activation functions. cf f represents the feature encoding of the human body's frontal normal image. cb F represents the feature encoding of the human back normal image. cat (·) indicates the feature concatenation operation, F grid (·) indicates grid sampling interpolation.

[0028] Among them, the human posture feature encoding module extracts local feature encoding f lo and volume feature encoding f v The specific steps are as follows:

[0029] The process involves calculating the signed distance function of the human body parameterized model, obtaining its signed distance function encoding, voxelizing the model, obtaining the 3D feature encoding of the voxelized model using 3D residual units, and then using mesh sampling interpolation to obtain local feature encoding based on the signed distance function encoding and extracting volumetric feature encoding based on the 3D feature encoding.

[0030]

[0031] Among them, F sdf (·) indicates the calculation of the human parametric model M. smp The operation of the signed distance function, F vol (·) indicates the voxelization operation of the mesh model, E 3d (·) indicates that the voxelized human parameterized model M is obtained through three-dimensional residual units. smp The operation of three-dimensional feature encoding, F grid (·) represents grid sampling interpolation, f lo f represents local feature encoding. v This represents the volume feature encoding.

[0032] Among them, the feature decoding reconstructs a colorless, textureless 3D human body model Y. s The output processing steps are as follows:

[0033] The human body surface normal map feature f i Local feature encoding f lo and volume feature encoding f v The values ​​are fed together into the feature decoder to obtain the symbolic distance function s of the 3D model. dA colorless 3D human body model Y is reconstructed using the moving cube algorithm based on the signed distance function value. s :

[0034]

[0035] Among them, F cat (·) indicates a feature concatenation operation, E g (·) denotes the operation of the graph convolutional decoder, which consists of graph convolutional layers, one-dimensional convolutional layers, batch normalization, and ReLU activation functions. d F represents the value of the signed distance function. mc (·) represents the operation of the moving cube algorithm, Y s This represents a 3D model without color or texture.

[0036] The back image generation module generates a human back image x. b The processing steps are as follows:

[0037] X i and the normal diagram of the back of the human body N cb The human back image is obtained by concatenating and encoding features, which are then fed into an image encoding unit for further encoding. This results in the human back image being encoded, and subsequently fed into a channel attention residual unit for further encoding. After upsampling, the human back image (x) is obtained. b :

[0038]

[0039] Among them, f bf E represents the encoded features of the human back. s (·) represents the operation of the image encoder, which consists of convolutional layers, instance normalization, and ReLU activation function. F cat (·) indicates feature concatenation, E res (·) denotes the operation of the channel attention residual unit, which consists of a convolutional layer, a channel attention layer, batch normalization, and a ReLU activation function. F up (·) indicates the upsampling operation, which consists of a transposed convolutional layer, batch normalization, and a ReLU activation function.

[0040] The texture mapping process includes the following steps:

[0041] Using grid sampling interpolation values ​​for human body image X i Extract the color code of the human body model, rasterize the human body model color code through a mesh model, and select non-repeating values ​​to obtain the visibility of the human body model; based on the human body 3D model Y without color texture. sA 3D model with frontal color texture is obtained by utilizing human body model color encoding and human body model visibility; mesh sampling interpolation is used to model the back of the human body image X. b The color code of the human back model is extracted, and then rasterized through a mesh model. The visibility of the human back model is obtained by selecting non-repeating values. Based on the 3D model with frontal colors, the color code of the human back model, and the visibility of the human back model, the human image X is used... i Image X of the back of the human body b Texture mapping is performed to obtain a 3D human body model Y with front and back color textures. c :

[0042]

[0043] Among them, F grid (·) represents grid sampling interpolation, f fc Indicates the color code of the frontal human body model, F ras (·) indicates rasterization of the mesh model, F u (·) indicates the operation of selecting unique values, f fv Y represents the visibility of a frontal human model. cf f represents a 3D model with positive color texture. bc Indicates the color code of the human back model, f bv Indicates the visibility of the human back model, Y c This represents a 3D model with front and back color textures.

[0044] The processing steps of the attitude optimization module are as follows:

[0045] The pose features are fed into a linear layer to obtain high-dimensional pose features. The KNN algorithm is used to construct the pose feature map structure corresponding to the high-dimensional pose features. The pose feature map structure is then fed into a graph convolutional unit and a linear layer to obtain optimized pose features. The optimized pose features are calculated using the Rodrigues rotation formula.

[0046]

[0047] Among them, f s E represents the posture feature. l (·) represents the operation of the linear layer, F knn (·) represents the operation of the KNN algorithm, g f Represents the pose feature map structure, E g (·) denotes the operation of a graph convolution unit, f gs f represents the pose optimization feature. rs F represents the optimized pose features.rod (·) represents the Rodriguez rotation operation, with the specific formula as follows:

[0048] F rod (r)=cosθI+(1-cosθ)nn T +sinθn^

[0049] Where I is the identity matrix, n is the unit vector of vector r, θ is the magnitude of vector r, and n T Let n denote the transpose of n, and n^ denote the antisymmetric matrix of n.

[0050] The processing steps of the human image encoding module are as follows:

[0051] The human image encoding module encodes human images X. i Extracting image features f r Image features f r With human body image X i Perform grid sampling interpolation and feature concatenation separately to obtain the image feature code f. c :

[0052]

[0053] Among them, E c (·) indicates the operation of the human image coding module, which consists of stacked network units, residual units, and convolutional layers. The stacked network units include pooling layers, graph convolutional blocks, and upsampling layers. The graph convolutional blocks include graph convolutional layers, linear layers, and ReLU activation functions.

[0054] The method of this invention proposes a pose fusion 3D human body reconstruction network and a graph neural radiation field 3D human body reconstruction network. The pose fusion 3D human body reconstruction network reconstructs a 360-degree viewpoint mesh-form 3D human body model, and the graph neural radiation field 3D human body reconstruction network reconstructs a human neural radiation field of the required accuracy from the human body image, and renders an image of the human body at any angle.

[0055] This invention considers both local and global features, and feeds the fused features into the feature decoding and reconstruction module to reconstruct the three-dimensional human body shape model. The back of the human body is generated from the human body image and the surface normal map. Texture mapping is performed using the human body image and the back of the human body image, which achieves good shape reconstruction and color texture reconstruction effects, and further improves the reconstruction quality of the three-dimensional human body model. Attached Figure Description

[0056] Figure 1 This is a flowchart of the three-dimensional reconstruction method according to an embodiment of the present invention;

[0057] Figure 2This is a structural diagram of the surface normal map generation module according to an embodiment of the present invention;

[0058] Figure 3 This is a structural diagram of the surface normal map feature encoding module according to an embodiment of the present invention;

[0059] Figure 4 This is a structural diagram of the feature decoding and reconstruction module and the texture mapping module according to an embodiment of the present invention;

[0060] Figure 5 This is a structural diagram of the human image encoding module according to an embodiment of the present invention. Detailed Implementation

[0061] The present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments. It should be understood that the specific embodiments described herein are only for explaining the present invention and are not intended to limit the present invention.

[0062] The image-based 3D human body reconstruction method of this invention is implemented by a pose fusion 3D human body reconstruction network and a graph neural radiation field 3D human body reconstruction network. The input condition for the pose fusion 3D human body reconstruction network is a single image or at least 4 images. The pose fusion 3D human body reconstruction network reconstructs a 360-degree viewpoint mesh-like 3D human body model. The graph neural radiation field 3D human body reconstruction network reconstructs a human neural radiation field with the required accuracy from human body images (e.g., more than 500 images) and renders an image of the human body at any angle.

[0063] The posture fusion 3D human reconstruction network includes a surface normal map generation module, a surface normal map feature encoding module, a human posture feature encoding module, a feature decoding and reconstruction module, a back image generation module, and a texture mapping module. The graph neural radiation field 3D human reconstruction network includes a human image encoding module, a posture optimization module, a feature decoding module, and an integration module.

[0064] When the posture fusion 3D human reconstruction network reconstructs, it first obtains the surface normal map through the surface normal map generation module, obtains the surface normal map feature encoding module, extracts the local feature encoding and volume feature encoding from the human parametric model, sends the fused features to the feature decoding reconstruction module to reconstruct the human shape 3D model, obtains the human back image from the human image and the surface normal map, and then performs texture mapping based on the texture mapping module using the human image and the human back image.

[0065] During the reconstruction of the three-dimensional human body using the neural radiation field network, the image features of the human body are first extracted by the human body image encoding module. These features are then interpolated using grid sampling and feature stitching to obtain image feature encoding. The pixels of the human body image are converted into three-dimensional points in the neural radiation field. Position encoding is added to these three-dimensional points, and the pose features are sent to the pose optimization module to obtain the optimized pose encoding. The orientation encoding of the three-dimensional points is calculated using camera parameters and sent along with the image feature encoding, pose encoding, and three-dimensional point features to the feature decoding module to obtain color and shape features. Finally, the signed distance function value and shape features are sent to the integration module to obtain the volume density value, and a new perspective image of the human body is rendered and output.

[0066] This invention employs a surface normal map feature encoding module for feature extraction. It uses a stacked network to encode the human body surface normal map, increasing the network depth and fusing features from different stages. In the residual encoding, a channel attention layer is added, and a Transformer network is used to capture long-distance dependencies, thereby obtaining the relationships between different parts of the human body surface normal map and improving the efficiency of feature extraction.

[0067] This invention employs a human posture feature encoding module to extract local feature codes and volume feature codes from a human parametric model, calculates the signed distance function of the human parametric model to obtain local feature codes, voxels the human parametric model, obtains volume feature codes from the human parametric model through three-dimensional residual encoding, and combines the local feature codes and volume feature codes of the human parametric model to reconstruct a human three-dimensional model with high precision and completeness.

[0068] This invention employs a feature decoding and reconstruction module for feature decoding. The module feeds the human body surface normal graph features, local feature codes, and volume feature codes into the feature decoder to obtain the signed distance function value of the human 3D model. A colorless, textureless human 3D model is then reconstructed using the moving cubes algorithm. The feature decoder consists of graph convolutional coding and one-dimensional convolutional coding. It utilizes the KNN algorithm to construct a graph structure of 3D points, obtaining the relationships between adjacent 3D points for more efficient feature decoding.

[0069] This invention employs a texture mapping module, uses mesh sampling interpolation to extract the color code of the human body model, rasterizes the mesh model, selects non-repeating values ​​to obtain the visibility of the human body model, uses the color code and visibility of the human body model to obtain the front and back color textures of the 3D model, and simultaneously uses the human body image and the back image of the human body to perform texture mapping, thereby improving the completeness of the reconstructed texture and obtaining a 3D human body model with front and back color textures.

[0070] This invention employs a pose optimization module, which uses the KNN algorithm to construct a pose feature map structure corresponding to high-dimensional pose features. The pose feature map structure is then fed into graph convolutional encoding to obtain high-dimensional pose optimization features, thereby improving the pose optimization effect.

[0071] This invention employs a human image encoding module, using a stacked network to encode human images, increasing the network depth and fusing features from different stages. It divides image features into multiple parts and uses the KNN algorithm to construct a graph structure of image features, obtaining the relationships between different parts of the human image, improving the efficiency of feature extraction, providing more accurate human image features for generating images from new perspectives, and improving the effect of generating images from new perspectives.

[0072] The overall process of the human body three-dimensional reconstruction method proposed in this invention is as follows: Figure 1 As shown, the process of the pose fusion 3D human reconstruction network includes a surface normal map generation module, a surface normal feature encoding module, a human pose feature encoding module, a feature decoding and reconstruction module, and a texture mapping module; the process of the graph neural radiation field 3D human reconstruction network includes a pose optimization module, a human image encoding module, a feature decoding module, and an integration module.

[0073] From human image X i Predicting the human parameter model M using posture estimation methods smp The vector map N of the frontal normal of the human body parameterized model is obtained by feeding it into a differentiable renderer. sf and the back normal map N of the human body parameterized model sb , with human body image X i The data are fed into the surface normal map generation module to obtain the frontal normal map N of the human body. cf and the normal diagram of the back of the human body N cb The surface normal map feature encoding module is used to extract the human body surface normal map features f. i ; The human body parameterization model M smp The data is fed into the human posture feature encoding module to extract local feature codes f. lo and volume feature encoding f v , and the characteristics of the human body surface normal map f i The values ​​are fed together into the feature decoding and reconstruction module to obtain the symbolic distance function value s. d And reconstruct a colorless, textureless 3D human body model Y s Output: Human body image X i and the normal diagram of the back of the human body N cb The images are fed together into the back image generation module to generate a human back image X. b Human body image X i Image of the back of the human body and a 3D model of the human body without color texture Y sThe data are fed into the texture mapping module to reconstruct a 3D human body model Y with front and back color textures. c Output.

[0074] Human image X extracted by human image encoding module i Image features f r Image features f r With human body image X i Perform grid sampling interpolation and feature concatenation separately to obtain the image feature code f. ca ; X-ray the human body image i The pixels are converted into three-dimensional point coordinates p in the neural radiation field, and positional encoding is added to obtain three-dimensional point features f with positional encoding. l and pose features f s The optimized attitude code f is obtained by sending it to the attitude optimization module. rs The orientation code f of the 3D point is calculated from the camera parameters. v , and image feature encoding f ca Attitude coding f rs and three-dimensional point features f l The two data points are fed into the feature decoding module to obtain the color value c and the shape feature f. σ The symbolic distance function value s d and shape features f σ The data are fed into the integrated module to obtain the volume density value σ, and then rendered to output a new perspective image Y of the human body corresponding to the camera parameters. n .

[0075] The surface normal map generation module designed in this invention generates surface normal maps more efficiently from image X. i Predicting the human parameter model M using posture estimation methods smp The vector map N of the frontal normal of the human body parameterized model is obtained by feeding it into a differentiable renderer. sf and the back normal map N of the human body parameterized model sb :

[0076]

[0077] Among them, G s (·) denotes the attitude estimation operation, R f (·),R b (·) indicates the operation of rendering the front and back normal maps of the human body parametric model. Then, the front image of the human body, the front normal map of the human body parametric model, and the back normal map of the human body parametric model are fed into the surface normal map generation module to obtain the front normal map N of the human body. cf and the normal diagram of the back of the human body N cb :

[0078]

[0079] Among them, f sf ,f sb E represents the frontal and back normal map encoding features of the human body parameterized model, respectively. s (·) represents the encoding operation of the image encoder, which consists of a convolutional layer, instance normalization, and a ReLU activation function. F cat (·) indicates a feature concatenation operation, E res (·) indicates the encoding operation of the residual encoder, F up The dot (·) indicates an upsampling operation. An upsampling layer consists of a transposed convolutional layer, batch normalization, and a ReLU activation function. See also... Figure 2 As shown, the surface normal map generation module consists of a convolutional layer, a residual unit 1, a residual unit 2, a residual unit 3, a residual unit 4, a residual unit 5, a residual unit 6, a residual unit 7, a residual unit 8, a residual unit 9, and an upsampling layer. The residual units are composed of a Conv+ReLU layer, a Conv+ReLU layer, a Conv layer, a max pooling layer, a Conv+ReLU layer, a Conv layer, and a Sigmoid activation layer. The output of the previous Conv layer is used as the input of the max pooling layer, and is added to the output of the Sigmoid activation layer. This sum, along with the input of the residual unit, becomes the output of the residual unit.

[0080] The surface normal map feature encoding module designed in this invention encodes the frontal and back normal maps of the human body respectively, obtaining the feature codes for the frontal and back normal maps. After concatenating the feature codes, grid sampling interpolation is performed to obtain the human body surface normal map feature f. i :

[0081]

[0082] Among them, E c(·) represents the human surface normal map encoding operation of the surface normal map feature encoding module. The surface normal map feature encoding module consists of stacked network units, residual units, and convolutional layers. The stacked network units contain a multi-level structure of max pooling layers (AvgPool), Transformer network encoding layers, and upsampling layers (upsamplpe). Specifically, it could have a four-layer structure of max pooling layers, three upsampling layers, and five Transformer network encoding modules. The max pooling layers are sequentially connected, and the last max pooling layer is connected to the first upsampling layer of the three sequentially connected upsampling layers via a Transformer network encoding module. The input of each other Transformer network encoding module is the output of the previous max pooling layer. The input of the last Transformer network encoding module is the input of the stacked network units, and the input of each other Transformer network encoding module is the output of the previous max pooling layer. The output of the rmer network encoding module serves as the input to each upsampling layer, and the output of the last Transformer network encoding module and the output of the last upsampling layer serve as the output of the stacked network unit. The Transformer network encoding module sequentially includes a layer normalization layer, a multi-head attention layer (W-MSA), a layer normalization layer, and a multilayer perceptron layer (MLP). The input of the Transformer network encoding layer and the output of the W-MSA layer serve as the input to the next layer normalization layer, and the input of the next layer normalization layer and the output of the MLP layer serve as the output of the rmer network encoding layer. The residual unit sequentially includes a Con+ReLU layer, a Con+ReLU layer, and a convolutional layer (Con). The input of the residual unit and the output of its convolutional layer (Con) serve as the input to the convolutional layer adjacent to the residual unit in the surface normal map feature encoding module. cf f represents the feature encoding of the human body's frontal normal image. cb F represents the feature encoding of the human back normal image. cat (·) indicates the feature concatenation operation, F grid (·) indicates grid sampling interpolation. See also Figure 3 As shown, the surface normal map feature encoding module consists of stacked network units - residual units - convolutional layers - convolutional layers - convolutional layers - stacked network units - residual units - convolutional layers in sequence. The input of the second stacked network unit includes the output of the convolutional layer above it and the output of the convolutional layer above it.

[0083] See Figure 1As shown, the human posture feature encoding module of the present invention includes a symbolic distance calculation unit, a model voxelization unit, and a three-dimensional residual unit. The module extracts local feature codes and volumetric feature codes from the human parametric model, calculates the symbolic distance function of the human parametric model to obtain its symbolic distance function code, voxels the human parametric model, obtains the three-dimensional feature code of the human parametric model through the three-dimensional residual unit, and performs mesh sampling interpolation on the symbolic distance function code and the three-dimensional feature code to obtain the local feature code and volumetric feature code, respectively.

[0084]

[0085] Among them, F sdf (·) represents the operation of calculating the signed distance function of the human parametric model, which is processed by the signed distance calculation unit, F vol (·) indicates the voxelization operation of the mesh model, which is performed by the model voxelization unit. E 3d (·) represents the operation of a 3D residual unit, which consists of a 3D convolutional layer, a 3D batch normalization function, and a ReLU activation function. F grid (·) represents grid sampling interpolation, f lo f represents local feature encoding. v This represents the volume feature encoding.

[0086] The feature decoding and reconstruction module designed in this invention feeds human body surface normal map features, local feature encoding, and volume feature encoding into the feature decoder to obtain the signed distance function value of the 3D model, and reconstructs a colorless 3D human body model using the moving cube algorithm.

[0087]

[0088] Among them, F cat (·) indicates a feature concatenation operation, E g (·) denotes the operation of the graph convolutional decoder, which consists of graph convolutional layers, one-dimensional convolutional layers, batch normalization, and ReLU activation functions. d F represents the value of the signed distance function. mc (·) represents the operation of the moving cube algorithm, Y s This represents a 3D model without color or texture.

[0089] See Figure 4As shown, the feature decoding and reconstruction module designed in this invention consists of a graph convolutional block, a graph convolutional block, a linear layer, and a linear layer. The output of the lower-level graph convolutional block serves as the input of two linear layers, and the input of the lower-level linear layer also includes the output of the upper-level linear layer. The graph convolutional block consists of a graph convolutional layer (GCN), a ReLU activation layer, a linear layer (Linear), and a ReLU activation layer. The input of the linear layer (Linear) includes the output of the previous ReLU activation layer and the input of the graph convolutional block or the graph convolutional layer (GCN). The input of the linear layer (Linear) and the output of the next-level ReLU activation layer serve as the output of the graph convolutional block.

[0090] The back image generation module designed in this invention concatenates and encodes a human body image and a human back normal map to obtain the encoded features of the human back, which are then fed into channel attention residual encoding to obtain the human back image x. b :

[0091]

[0092] Among them, f bf E represents the encoded features of the human back. s (·) represents the operation of the image encoder, which consists of convolutional layers, instance normalization, and the ReLU activation function, F cat (·) indicates feature concatenation, E res (·) represents the operation of the residual unit, which consists of a convolutional layer, a channel attention layer, batch normalization, and a ReLU activation function, F up The dot (·) indicates an upsampling operation. An upsampling layer consists of a transposed convolutional layer, batch normalization, and a ReLU activation function. See also... Figure 4 As shown, the back image generation module consists of a convolutional layer, a ReLU activation layer, a residual unit, and an upsampling layer connected in sequence. The input image is added to the back normal map of the human body and then used as the input to the convolutional layer of the back image generation module.

[0093] The texture mapping module designed in this invention uses mesh sampling interpolation to extract the color code of the human body model, rasterizes the mesh model, selects non-repeating values ​​to obtain the visibility of the human body model, and uses the human body model color code and visibility to obtain the front and back color textures of the 3D model. At the same time, it uses the human body image and the back image of the human body to perform texture mapping, improves the completeness of the reconstructed texture, and obtains a 3D human body model with front and back color textures.

[0094]

[0095] Among them, F grid (·) represents grid sampling interpolation, f fc Indicates the color code of the frontal human body model, Fras (·) indicates rasterization of the mesh model, F u (·) indicates the operation of selecting unique values, f fv Y represents the visibility of a frontal human model. cf f represents a 3D model with positive color texture. bc Indicates the color code of the human back model, f bv Indicates the visibility of the human back model, Y c This represents a 3D model with front and back color textures.

[0096] The pose optimization module designed in this invention includes linear layer units and graph convolutional units. The output of the linear layer unit serves as the input of the graph convolutional unit. Pose features are fed into the linear layer unit to obtain higher-dimensional pose features. The KNN algorithm is used to construct the pose feature map structure corresponding to the higher-dimensional pose features. The pose feature map structure is then fed into the graph convolutional unit for convolution processing before being fed into the linear layer unit for further processing to obtain the optimized pose features. The optimized pose features are calculated using the Rodrigues rotation formula.

[0097]

[0098] Among them, f s E represents the posture feature. l (·) represents the operation of a linear layer unit, F knn (·) represents the operation of the KNN algorithm, g f Represents the pose feature map structure, E g (·) denotes the operation of a graph convolutional unit, which can consist of a graph convolutional layer, batch normalization, and a ReLU activation function. gs f represents the pose optimization feature. gs F represents the optimized pose features. rod (·) represents the Rodriguez rotation operation, with the specific formula as follows:

[0099] F rod (r)=cosθI+(1-cosθ)nn T +sinθn^

[0100] Where I is the identity matrix, n is the unit vector of vector r, θ is the magnitude of vector r, and n T Let n denote the transpose of n, and n^ denote the antisymmetric matrix of n.

[0101] The human image encoding module designed in this invention encodes human image X. i The image is fed into the human body image encoding module to extract image features f rThen, the image features and the human body image are subjected to grid sampling interpolation and feature concatenation to obtain the image feature code f. c :

[0102]

[0103] Among them, E c (·) indicates the encoding operation of the human image encoding module, which consists of stacked network units, residual units, and convolutional layers. See [link to documentation]. Figure 5 As shown, its network structure is as follows: stacked network unit - residual unit - convolutional layer - convolutional layer - convolutional layer - stacked network unit - residual unit - convolutional layer. The input of the second stacked network unit comes from the output of the previous convolutional layer and the output of the convolutional layer before that. The stacked network unit includes a pooling layer, a graph convolutional block, and an upsampling layer. The pooling layer and the upsampling layer are multi-level structures, such as four layers. There are multiple graph convolutional blocks. The output of one pooling layer corresponds to the input of one graph convolutional block, and the output of one graph convolutional block corresponds to the input of one upsampling layer. The input of the next-level upsampling layer also includes the output of the previous-level upsampling layer. The input of the next-level pooling layer also includes the output of the previous-level pooling layer. The input of the first upsampling layer includes the output of two graph convolutional blocks. After the input of the stacked network unit passes through a graph convolutional block, its output is added to the output of the last-level upsampling layer to become the output of the stacked network unit. The graph convolutional block sequentially includes a graph convolutional layer, a ReLU activation function, a linear layer, and another ReLU activation function. The input of the graph convolutional layer and the output of the previous ReLU activation function serve as the input of its linear layer, and the input of its linear layer and the output of the next ReLU activation function serve as the output of the graph convolutional block. The residual unit sequentially includes a convolutional layer Con + a ReLU activation layer - a convolutional layer Con + a ReLU activation layer - a convolutional layer Con. The input of the residual unit is the output of the stacked network units. The output of the convolutional layer Con of the residual unit and the input of the residual unit serve as the input of the convolutional layer of the human image coding module that receives the output of the residual unit.

[0104] The above description is only a preferred embodiment of the present invention. It should be noted that, for those skilled in the art, several improvements and modifications can be made without departing from the principle of the present invention, and these improvements and modifications should also be considered within the scope of protection of the present invention.

Claims

1. An image-based method for three-dimensional human body reconstruction, characterized in that, The input image is processed and reconstructed separately by a pose fusion 3D human reconstruction network and a graph neural radiation field 3D human reconstruction network, and then fused to achieve the final 3D reconstruction. The pose fusion 3D human reconstruction network includes a surface normal map generation module, a surface normal map feature encoding module, a human pose feature encoding module, a feature decoding and reconstruction module, a back view image generation module, and a texture mapping module. The graph neural radiation field 3D human reconstruction network includes a pose optimization module, a human image encoding module, a feature decoding module, and an integration module. The processing steps of the human 3D reconstruction method are as follows: From human body images Predicting human parameterized models using pose estimation methods The vector map is fed into a differentiable renderer to obtain the frontal normal map of the human body parameterized model. and the back normal plot of the human body parametric model With human body images The data are fed into the surface normal map generation module to obtain the frontal normal map of the human body. and the normal diagram of the back of the human body The surface normal map features are then fed into the surface normal map feature encoding module to extract the surface normal map features of the human body. Parametric model of the human body The data is fed into the human posture feature encoding module to extract local feature codes. and volume feature encoding Features of the normal map of the human body surface The values ​​are fed together into the feature decoding and reconstruction module to obtain the symbolic distance function value. And reconstruct a colorless, textureless 3D human body model. Output: Human body image and the normal diagram of the back of the human body The images are fed together into the back image generation module to generate a human back image. Human body images Images of the back of the human body and colorless textured 3D human body model The data are fed into the texture mapping module to reconstruct a 3D human body model with front and back color textures. Output; Human images are extracted by the human image encoding module. Image features Image features With human body images Perform grid sampling interpolation and feature concatenation separately to obtain image feature encoding. Human body image Pixels are converted into three-dimensional point coordinates in the neural radiation field. Add positional encoding to obtain 3D point features with positional encoding. and pose features The optimized attitude code is sent to the attitude optimization module to obtain the optimized attitude code. The orientation code of a 3D point is calculated from the camera parameters. Image feature encoding Posture characteristics and 3D point features The values ​​are sent together to the feature decoding module to obtain the color values. and shape features ; the symbolic distance function value and shape features The samples are fed together into the integrated module to obtain the volume density value. Render and output an image of the human body from a preset viewpoint corresponding to the camera parameters. ; The surface normal map generation module obtains the frontal normal map of the human body. and the normal diagram of the back of the human body The steps are as follows: Human body image Frontal normal map of human body parametric model After feature concatenation, the data is fed into an image encoder to obtain the encoded features representing the frontal normal map of the human body parameterized model. human body image With the back normal plot of the human body parametric model After feature concatenation, the data is fed into an image encoder to obtain the encoded features of the back face normal map of the human body parameterized model. Encoding features of the frontal normal map and the back normal map of the human body parametric model. , After being processed by the residual encoder, the images are upsampled to obtain the frontal normal image of the human body. and the normal diagram of the back of the human body ; indicates the following: ； in, , These represent the frontal normal map encoding features and the back normal map encoding features of the human body parametric model, respectively. This represents the encoding operation of the image encoder. This indicates a feature concatenation operation. This indicates the operation of the residual encoder. Indicates an upsampling operation; The back image generation module generates a human back image. The processing steps are as follows: Human body image and the normal diagram of the back of the human body The back of the human body is obtained by concatenation encoding, which is then fed into an image encoder for encoding to form the back of the human body encoded features. These features are then fed into a channel attention residual unit for encoding, and finally upsampled to obtain the back of the human body image. : ； in, Indicates the coding features on the back of the human body. This indicates the operation of the image encoder. This indicates the operation of feature concatenation. This represents the operation of the channel attention residual unit. This indicates an upsampling operation; The processing steps of the attitude optimization module are as follows: The pose features are fed into a linear layer to obtain high-dimensional pose features, which are then used... KNN The algorithm constructs a pose feature map structure corresponding to high-dimensional pose features, feeds the pose feature map structure into a graph convolutional unit for processing, and then feeds it into a linear layer for further processing to obtain optimized pose features. The optimized pose features are then calculated using the Rodrigues rotation formula. ； in, Indicates posture characteristics, Represents linear layer operations. express KNN Algorithm operations, Represents the structure of the pose feature map. This represents the operation of a graph convolution unit. This represents pose optimization features. This represents the optimized pose features. The Rodriguez rotation operation is represented by the following formula: ； in, It is the identity matrix. It is a vector unit vector, It is a vector The length of the mold, express The transpose of the matrix, express An antisymmetric matrix.

2. The image-based three-dimensional human body reconstruction method according to claim 1, characterized in that, The surface normal map feature encoding module extracts the human body surface normal map features. The steps are as follows: Normal diagram of the front of the human body and the normal diagram of the back of the human body Encoding is performed to obtain the feature codes of the human body's frontal and back normal maps. These feature codes are then concatenated and interpolated using grid sampling to obtain the human body surface normal map features. : ； in, This represents the human body surface normal map encoding operation of the surface normal map feature encoding module. This represents the encoding of the human body's frontal normal image features. This represents the encoding of the normal map features of the human back. This indicates the operation of feature concatenation. This indicates grid sampling interpolation.

3. The image-based three-dimensional human body reconstruction method according to claim 2, characterized in that, The human posture feature encoding module extracts local feature encoding. and volume feature encoding The steps are as follows: Computation of human parametric models The signed distance function is used to obtain the parametric model of the human body. Symbolic distance function encoding Parametric model of the human body Voxelization is performed, and a voxelized parametric human body model is obtained through three-dimensional residual units. The three-dimensional feature encoding is based on the signed distance function through grid sampling interpolation. Obtaining local feature encoding And volume feature encoding based on 3D feature encoding. : ； in, Represents the computation of human parametric models Operations of the signed distance function, This represents the operation of voxelizing the mesh model. This indicates that a voxelized parametric human body model is obtained through three-dimensional residual units. The operation of three-dimensional feature encoding, Indicates grid sampling interpolation, Represents local feature encoding. This represents the volume feature encoding.

4. The image-based three-dimensional human body reconstruction method according to claim 3, characterized in that, The feature decoding and reconstruction module reconstructs a colorless, textureless 3D human body model. The output processing steps are as follows: Human surface normal map features Local feature coding and volume feature encoding After feature concatenation, the data is fed into a graph convolutional decoder to obtain the signed distance function value of the 3D model. A colorless 3D human body model was reconstructed using the moving cube algorithm based on the signed distance function value. : ； in, This indicates a feature concatenation operation. This represents the operation of the graph convolution decoder. Represents the value of the signed distance function. This represents the operation of the moving cube algorithm. This represents a 3D human body model without color or texture.

5. The image-based three-dimensional human body reconstruction method according to claim 4, characterized in that, The processing steps of the texture mapping module are as follows: Using grid sampling interpolation on human images Extract color codes from the human body model, rasterize the color codes using a mesh model, and select non-repeating values ​​to obtain the visibility of the human body model; based on a 3D human body model without color texture. A 3D model with frontal color is obtained by utilizing human body model color encoding and human body model visibility; mesh sampling interpolation is used to interpolate the back of the human body image. The color code of the human back model is extracted, rasterized using a mesh model, and the visibility of the human back model is obtained by selecting non-repeating values. Based on the 3D model with frontal colors, the color code of the human back model, and the visibility of the human back model, the human image is used... Images of the back of the human body Texture mapping is performed to obtain a 3D human body model with front and back color textures. : ； in, Indicates grid sampling interpolation, This indicates the color code for a frontal human body model. Indicates the rasterization of the mesh model. This indicates the operation of selecting unique values. Indicates the visibility of a frontal human body model. This represents a 3D model with positive color texture. This indicates the color code of the human back model. Indicates the visibility of the human back model. This represents a 3D model with front and back color textures.

6. The image-based three-dimensional human body reconstruction method according to claim 5, characterized in that, The processing steps of the human body image encoding module are as follows: Human image encoding module for human images Extracting image features Image features With human body images Perform grid sampling interpolation and feature concatenation separately to obtain image feature encoding. : ； in, This indicates the operation of the human body image encoding module. This indicates grid sampling interpolation.

Images