A 3D face construction method

Abstract

This invention discloses a 3D face construction method, relating to the field of bionics. Specifically, it includes the following steps: acquiring feature vectors from multiple viewpoint images of a face and fusing them to generate a fused feature vector; generating a 3D face shape model based on the fused feature vector; and applying texture filling to the 3D shape model to generate the face model. The aim is to improve the accuracy of digital face construction.

Description

Technical Field

[0001] This invention relates to the field of bionics, and in particular to a method for constructing a 3D human face. Background Technology

[0002] 3D modeling is a fundamental research area in computer graphics processing, with typical applications including 3D games, visual communication systems, virtual reality augmentation, and medical simulation; it can be referred to as digital human. Especially with the increase in network bandwidth and the emergence of MPEG-4 face encoding systems for real-time, ultra-low bitrate visual applications, 3D face modeling has become a research hotspot in video telephony and virtual web conferencing applications. However, rapid 3D face modeling remains a highly challenging topic. Due to its wide range of applications, it has attracted numerous researchers to conduct extensive research on 3D face modeling, yielding fruitful results. For example, the paper "Rapid and Automatic 3D Face Modeling Using Active Appearance Models" proposes an improved method for active appearance models during face feature extraction, utilizing initial position and instance model selection. Simultaneously, it uses eigenfaces and spatial affine transformation rules to adjust the face model, achieving matching of the corresponding face's position and shape. This imaging method is simple, but its imaging accuracy is low, with local ghosting appearing, making it only suitable for applications with low imaging requirements.

[0003] Therefore, improving the accuracy of digital human face reconstruction has become an urgent technical challenge. Summary of the Invention

[0004] The main objective of this invention is to provide a 3D face construction method, which aims to improve the accuracy of digital face construction.

[0005] To achieve the above objectives, this invention proposes a 3D face construction method, comprising the following steps:

[0006] The feature vectors of multiple viewpoint images of a face are obtained and fused to generate a fused feature vector;

[0007] A three-dimensional face shape model is generated based on the fused feature vectors;

[0008] The three-dimensional shape model is textured to generate a face model.

[0009] In one embodiment of this application, obtaining feature vectors from multiple viewpoint images of a face and fusing them to generate a fused feature vector includes:

[0010] Acquire facial data of the same person from multiple different perspectives;

[0011] Each face data is encoded by a deep convolutional neural network with shared weights to obtain face feature vectors from each viewpoint;

[0012] Multiple facial feature vectors are concatenated and then fused using a multilayer perceptron network to generate a fused feature vector.

[0013] In one embodiment of this application, generating a three-dimensional face shape model based on the fused feature vector includes:

[0014] Obtain the fused feature vector;

[0015] The fused feature vector is decoded by a multilayer perceptron to obtain the head pose vector of the three-dimensional model;

[0016] The fused feature vector is decoded using a graph convolutional neural network to obtain a three-dimensional vertex vector; and

[0017] A 3D face shape model is generated based on the head pose vector and 3D vertex vector of the 3D model.

[0018] In one embodiment of this application, the head pose vector includes:

[0019] At least one of the following: the pitch angle of the face, the yaw angle of the face, the tilt angle of the face, and the displacement vector in three-dimensional space.

[0020] In one embodiment of this application, texture filling of the three-dimensional shape model includes:

[0021] Obtain the fused feature vector;

[0022] The fused feature vector is decoded using a large generative adversarial network to obtain facial texture parameters, and the obtained facial texture parameters are used to fill the three-dimensional shape model.

[0023] In one embodiment of this application, the method further includes calculating the similarity between any two viewpoint images in multiple viewpoint images of a face; when the similarity between any two viewpoint images does not meet a preset condition, the feature vectors of the two viewpoint images are not fused.

[0024] In one embodiment of this application, the formula for calculating the similarity between any two face images is as follows:

[0025]

[0026]

[0027]

[0028] Where l(x,y) represents the similarity between sample x and sample y in terms of brightness; c(x,y) represents the similarity between sample x and sample y in terms of contrast; s(x,y) represents the similarity between sample x and sample y in terms of structure; generally, c3 = c2 / 2 is taken;

[0029] Where, μ x μ y Let σ be the mean of sample x and sample y, respectively. x σ y Let σ be the variance of sample x and sample y, respectively. xy The covariance of sample x and sample y; the formulas for calculating the constants c1 and c2 are as follows:

[0030] c1 = (k1L) 2

[0031] c2=(k2L) 2

[0032] Where k1 defaults to 0.01, k2 defaults to 0.03, and L is the range of pixel values;

[0033] The Structural Similarity (SSIM) formula is as follows:

[0034] SSIM(x,y)=f[l(x,y)·c(x,y)·s(x,y)].

[0035] By adopting the above technical solution, feature vectors of multiple viewpoint images of a face are obtained, and then the feature vectors of multiple viewpoint images are fused to reduce the error caused by viewpoint differences and improve the accuracy of face texture and shape reconstruction. Attached Figure Description

[0036] The present invention will now be described in detail with reference to specific embodiments and accompanying drawings, wherein:

[0037] Figure 1 This is a schematic diagram of the structure of the first embodiment of the present invention. Detailed Implementation

[0038] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the following specific embodiments are only used to explain the invention and do not constitute a limitation thereof.

[0039] like Figure 1 As shown, in order to achieve the above objectives, this invention proposes a 3D face construction method, comprising the following steps:

[0040] The feature vectors of multiple viewpoint images of a face are obtained and fused to generate a fused feature vector;

[0041] A three-dimensional face shape model is generated based on the fused feature vectors;

[0042] The three-dimensional shape model is textured to generate a face model.

[0043] Specifically, a 3D face construction method includes the following steps: acquiring multiple viewpoint images of a face, which can be obtained using a VR3D instantaneous three-dimensional imaging system. Instantly obtaining omnidirectional photo information of the same object at the same moment can reduce the error between multiple viewpoint images of the face and correspondingly improve the similarity of multiple viewpoint images of the face, thereby facilitating the construction of the face model in the later stage.

[0044] Of course, it is conceivable that multiple perspective images of a face can also be captured in time-sharing using a regular optical camera.

[0045] To obtain feature vectors from multiple viewpoint images of a face, a neural convolutional network or a deep convolutional neural network with shared weights can be used.

[0046] Convolutional neural networks (CNNs) are a class of feedforward neural networks that incorporate convolutional computations and have a deep structure; they are one of the representative algorithms of deep learning. CNNs possess representation learning capabilities, enabling translation-invariant classification of input information according to their hierarchical structure. CNNs are constructed by mimicking biological visual perception mechanisms, allowing for both supervised and unsupervised learning. The shared parameters of the convolutional kernels within their hidden layers and the sparsity of inter-layer connections enable CNNs to learn gridded features, such as pixels and audio, with relatively low computational cost, achieving stable results and requiring no additional feature engineering of the data.

[0047] The input layers of convolutional neural networks can process multidimensional data. Typically, the input layer of a one-dimensional convolutional neural network receives a one-dimensional or two-dimensional array, where the one-dimensional array is usually a time or spectral sample; the two-dimensional array may contain multiple channels; the input layer of a two-dimensional convolutional neural network receives a two-dimensional or three-dimensional array; and the input layer of a three-dimensional convolutional neural network receives a four-dimensional array.

[0048] The hidden layers of a neural network include convolutional layers, pooling layers, and fully connected layers. The convolutional kernels in the convolutional layers contain weight coefficients. The function of a convolutional layer is to extract features from the input data. It contains multiple convolutional kernels, each element of which corresponds to a weight coefficient and a bias. Each neuron in a convolutional layer is connected to multiple neurons in a region located in the previous layer. The size of this region depends on the size of the convolutional kernel. When the convolutional kernel operates, it systematically scans the input features, performs element-wise matrix multiplication, sums the results, and adds the bias.

[0049] After feature extraction in the convolutional layers, the output feature maps are passed to pooling layers for feature selection and information filtering. The pooling layers contain predefined pooling functions that replace the result of a single point in the feature map with the statistics of the feature maps of its neighboring regions. The pooling layer selects the pooling region in the same way as the convolutional kernel scans the feature map, controlled by the pooling size, stride, and padding. Since feature vector extraction using neural convolutional networks is a conventional existing technique, its specific workflow will not be described in detail.

[0050] Feature vectors from multiple viewpoint images of a face are obtained using a neural convolutional network, and then these feature vectors are fused to generate a fused feature vector. By fusing feature vectors from multiple viewpoint images, errors caused by viewpoint differences can be significantly reduced. The fusion of feature vectors from multiple viewpoint images employs a multilayer perceptron (MPB). In this application, the MPB refers to a neural network consisting of fully connected layers with at least one hidden layer, and the output of each hidden layer is transformed by an activation function. The MPB enables rapid fusion of feature vectors from multiple viewpoint images while also reducing errors during the fusion process.

[0051] A 3D face shape model is generated based on the fused feature vector. The 3D face shape model in this application is constructed by pose vector and shape vector. Since the input parameter when constructing the 3D face shape model is the fused feature vector, the fused feature vector can significantly reduce the error caused by the difference in viewpoint, thereby directly improving the accuracy of face shape reconstruction.

[0052] After completing the 3D face shape model, texture filling is performed on the 3D face shape model. Since the input parameter for texture filling is a fused feature vector, the fused feature vector can significantly reduce the error caused by the difference in viewpoint, thus greatly improving the accuracy of texture filling.

[0053] The 3D face shape model in the application refers to a mathematical model, which can be used to simulate a 3D face framework on a display terminal. The face model in this application refers to a mathematical model, which can be used to simulate a face on a display terminal. The face model is obtained by combining a 3D face shape model with texture filling.

[0054] By adopting the above technical solution, feature vectors of multiple viewpoint images of a face are obtained, and then the feature vectors of multiple viewpoint images are fused to reduce the error caused by viewpoint differences and improve the accuracy of face texture and shape reconstruction.

[0055] In one embodiment of this application, obtaining feature vectors from multiple viewpoint images of a face and fusing them to generate a fused feature vector includes:

[0056] Acquire facial data of the same person from multiple different perspectives;

[0057] Each face data is encoded by a deep convolutional neural network with shared weights to obtain face feature vectors from each viewpoint;

[0058] Multiple facial feature vectors are concatenated and then fused using a multilayer perceptron network to generate a fused feature vector.

[0059] Specifically, obtaining feature vectors from multiple viewpoint images of a face and fusing them to generate a fused feature vector includes:

[0060] To obtain facial data from multiple different perspectives of the same face, it is conceivable that at least eight or more images of the face from multiple perspectives are needed to improve the accuracy of the facial model construction.

[0061] After acquiring data from multiple different perspectives of the same face, the face data is encoded using a deep convolutional neural network with shared weights. The deep convolutional neural network with shared weights in this application means that when the convolutional kernel is being calculated, the parameters of the filter in the convolutional kernel are fixed, and the scanning is completed by moving the surface according to a preset number of steps. All the scanned pixels share the weights in this filter for calculation.

[0062] Using a deep convolutional neural network with shared weights ensures the consistency of computational results. After encoding each face data point using this network, corresponding facial feature vectors from different viewpoints are obtained. These multiple feature vectors are then concatenated. Finally, a multilayer perceptron network is used to fuse these concatenated feature vectors, generating a fused feature vector.

[0063] The above technical solution encodes facial data from multiple viewpoints using a deep convolutional neural network encoder with shared weights, obtaining facial feature vectors for each viewpoint. These feature vectors from multiple viewpoints are then concatenated and fused using a multilayer perceptron network, providing input for shape and texture reconstruction. Fusing feature vectors from multiple viewpoints significantly reduces errors caused by viewpoint differences, improving the accuracy of facial texture and shape reconstruction.

[0064] In one embodiment of this application, generating a three-dimensional face shape model based on the fused feature vector includes:

[0065] Obtain the fused feature vector;

[0066] The fused feature vector is decoded by a multilayer perceptron to obtain the head pose vector of the three-dimensional model;

[0067] The fused feature vector is decoded using a graph convolutional neural network to obtain a three-dimensional vertex vector; and

[0068] A 3D face shape model is generated based on the head pose vector and 3D vertex vector of the 3D model.

[0069] Specifically, generating a 3D face shape model based on the fused feature vector includes: obtaining the fused feature vector; decoding the fused feature vector through a multilayer perceptron after obtaining the fused feature vector; obtaining the head pose vector of the 3D model after decoding; and predicting the head pose parameters of the 3D model through the head pose vector of the 3D model.

[0070] After obtaining the fused feature vector, a graph convolutional neural network is used to decode the fused feature vector to obtain the 3D vertex vector. Methods based on image 2D convolution result in a large number of parameters in deep networks, and because they do not fully utilize the local geometric features of the 3D face structure, this application uses a graph convolutional neural network to decode the fused feature vector. By relying on graph convolution to achieve information transfer between 3D vertices, it can fully utilize the local geometric features of the 3D face structure. This enables end-to-end 3D face model reconstruction and improves the accuracy of face model construction.

[0071] In one embodiment of this application, the head pose vector includes:

[0072] At least one of the following: the pitch angle of the face, the yaw angle of the face, the tilt angle of the face, and the displacement vector in three-dimensional space.

[0073] Specifically, the head pose vector includes: the pitch angle, yaw angle, and tilt angle of the face, as well as a three-dimensional displacement vector, encompassing six degrees of freedom of motion. This covers various perspectives of the face, improving the accuracy of face model construction.

[0074] In one embodiment of this application, texture filling of the three-dimensional shape model includes:

[0075] Obtain the fused feature vector;

[0076] The fused feature vector is decoded using a large generative adversarial network to obtain facial texture parameters, and the obtained facial texture parameters are used to fill the three-dimensional shape model.

[0077] Specifically, traditional 3D face texture reconstruction schemes represent face textures using low-dimensional texture parameters, resulting in insufficient realism and resolution. This application addresses this by obtaining fused feature vectors and decoding them using a large-scale generative adversarial network (GAN) to generate high-resolution face textures. The generated face textures, combined with a micro-renderer and face shape, can generate reconstructed face images from multiple perspectives. The generator network is trained using reconstruction loss and multi-view face discrimination loss. By employing a hinge loss function, batch normalization, and spectral normalization during network training, the training of the adversarial network is effectively stabilized, improving the resolution and quality of the generated face texture images.

[0078] Batch normalization in this application is a technique for training deep neural networks. It can not only speed up the convergence of the model, but more importantly, it can alleviate the problem of "gradient vanishing (dispersed feature distribution)" in deep networks to a certain extent, thereby making it easier and more stable to train deep network models.

[0079] In this application, spectral normalization refers to setting the total light intensity to one, so that the light intensity on the vertical axis is a decimal, which facilitates the training of adversarial networks.

[0080] In one embodiment of this application, the method further includes calculating the similarity between any two viewpoint images in multiple viewpoint images of a face; when the similarity between any two viewpoint images does not meet a preset condition, the feature vectors of the two viewpoint images are not fused.

[0081] Specifically, it calculates the similarity between any two viewpoint images of a face across multiple viewpoints. Structural similarity is a metric that measures the similarity between two given images, and its numerical value is determined based on three key features between the samples: brightness, contrast, and structure.

[0082] When the similarity between any two viewpoint images does not meet the preset conditions, it indicates that there is a significant difference between the two viewpoint images. In this case, the feature vectors of the two viewpoint images are not fused, thereby avoiding the amplification of the differences. The structure is simple and easy to implement.

[0083] In one embodiment of this application, the formula for calculating the similarity between any two face images is as follows:

[0084]

[0085]

[0086]

[0087] Where l(x,y) represents the similarity between sample x and sample y in terms of brightness; c(x,y) represents the similarity between sample x and sample y in terms of contrast; s(x,y) represents the similarity between sample x and sample y in terms of structure; generally, c3 = c2 / 2 is taken;

[0088] Where, μ x μ y Let σ be the mean of sample x and sample y, respectively. x σ y Let σ be the variance of sample x and sample y, respectively. xy The covariance of sample x and sample y; the formulas for calculating the constants c1 and c2 are as follows:

[0089] c1 = (k1L) 2

[0090] c2=(k2L) 2

[0091] Where k1 defaults to 0.01, k2 defaults to 0.03, and L is the range of pixel values;

[0092] The Structural Similarity (SSIM) formula is as follows:

[0093] SSIM(x,y)=f[l(x,y)·c(x,y)·s(x,y)].

[0094] The above description is merely a preferred embodiment of the present invention and does not limit the patent scope of the present invention. Any equivalent structural transformations made using the contents of the present invention's specification and drawings under the inventive concept of the present invention, or direct / indirect applications in other related technical fields, are included within the patent protection scope of the present invention.

Claims

1. A method for constructing a 3D face, characterized in that, Includes the following steps: The method involves acquiring feature vectors from multiple viewpoint images of a face and fusing them to generate a fused feature vector. Specifically, this involves acquiring face data from multiple different viewpoints of the same face; encoding each face data using a deep convolutional neural network with shared weights to obtain face feature vectors from each viewpoint; concatenating multiple face feature vectors and fusing them using a multilayer perceptron network to generate a fused feature vector. A 3D face shape model is generated based on the fused feature vector; specifically: the fused feature vector is obtained; the fused feature vector is decoded through a multilayer perceptron to obtain a 3D model head pose vector; the fused feature vector is decoded through a graph convolutional neural network to obtain a 3D vertex vector; and a 3D face shape model is generated based on the 3D model head pose vector and the 3D vertex vector. The three-dimensional face shape model is textured to generate a face model.

2. The 3D face construction method as described in claim 1, characterized in that, Head pose vectors include: At least one of the following: the pitch angle of the face, the yaw angle of the face, the tilt angle of the face, and the displacement vector in three-dimensional space.

3. The 3D face construction method as described in claim 1, characterized in that, Texture filling of the three-dimensional face shape model includes: Obtain the fused feature vector; The fused feature vector is decoded using a large generative adversarial network to obtain facial texture parameters, and the obtained facial texture parameters are used to fill in the 3D facial shape model.

4. The 3D face construction method as described in claim 1, characterized in that, It also includes calculating the similarity between any two viewpoint images in multiple viewpoint images of a face; when the similarity between any two viewpoint images does not meet the preset conditions, the feature vectors of the two viewpoint images are not fused.

5. The 3D face construction method as described in claim 4, characterized in that, The formula for calculating the similarity between any two face images is as follows: ； ； Where l(x,y) represents the similarity between sample x and sample y in terms of brightness; c(x,y) represents the similarity between sample x and sample y in terms of contrast; s(x,y) represents the similarity between sample x and sample y in terms of structure; generally, we take... ; in, Samples With sample The mean, Samples With sample variance sample With sample covariance; constant and The calculation formula is as follows: in, Default is , Default is , The range of pixel values; The Structural Similarity (SSIM) formula is as follows: 。

Images