Three-dimensional face modeling method and device, storage medium and electronic equipment

Abstract

The invention provides a three-dimensional face modeling method and device, a storage medium and electronic equipment, and the method comprises the steps: obtaining a feature extraction result of each two-dimensional face training image through an encoder network in an initial face generation model, so as to determine a to-be-generated vector of each two-dimensional face training image; calling a generator in the initial face generation model, and generating a three-dimensional face image corresponding to the corresponding two-dimensional face training image based on the to-be-generated vector of each two-dimensional face training image; determining a reflectivity image of a three-dimensional face image corresponding to each two-dimensional face training image so as to calculate a generation model loss value of the initial face generation model; and optimizing model parameters in the initial face generation model according to the direction of reducing the loss value of the generation model, so as to determine a target face generation model based on the initial face generation model after model optimization. According to the embodiment of the invention, three-dimensional face modeling can be conveniently carried out, and the accuracy of three-dimensional face modeling is improved.

Description

Technical Field

[0001] The present invention relates to the field of computer technology, and in particular to a three-dimensional face modeling method, device, storage medium and electronic equipment. Background Art

[0002] Currently, 3D face modeling has received extensive attention; however, related technologies usually achieve 3D face modeling through multi-view image fusion, but multiple cameras or special equipment may be required to obtain data, which increases the hardware cost and complexity of the system, and it is difficult to align images from multiple perspectives, resulting in low fusion effect and modeling accuracy. Based on this, there is currently no good solution for how to conveniently perform 3D face modeling and improve the accuracy of 3D face modeling. Summary of the invention

[0003] In view of this, the embodiments of the present invention provide a three-dimensional face modeling method, device, storage medium and electronic device to solve the problems that the related technology is relatively complex and the accuracy of three-dimensional face modeling is relatively low; that is, the embodiments of the present invention can construct (i.e., model) any two-dimensional face image into a corresponding three-dimensional face image through a target face generation model to realize convenient three-dimensional face modeling and effectively improve the accuracy of three-dimensional face modeling. In addition, the embodiments of the present invention do not require multiple cameras or special equipment support, which can effectively reduce the hardware cost and complexity of the system.

[0004] According to one aspect of an embodiment of the present invention, a 3D face modeling method is provided, the method comprising:

[0005] Acquire a face training data set, wherein the face training data set includes a plurality of two-dimensional face training images and a training image label corresponding to each of the plurality of two-dimensional face training images;

[0006] Calling the encoder network in the initial face generation model to perform feature extraction on each of the two-dimensional face training images to obtain feature extraction results of the corresponding two-dimensional face training images; and determining the vector to be generated of the corresponding two-dimensional face training image based on the feature extraction results of each of the two-dimensional face training images;

[0007] Calling the generator in the initial face generation model to generate three-dimensional face images corresponding to the corresponding two-dimensional face training images based on the vectors to be generated of the two-dimensional face training images respectively; and determining the reflectivity images of the three-dimensional face images corresponding to the two-dimensional face training images;

[0008] Calculating a generation model loss value of the initial face generation model based on the reflectance images of the three-dimensional face images corresponding to the two-dimensional face training images and the corresponding training image labels;

[0009] In the direction of reducing the loss value of the generation model, the model parameters in the initial face generation model are optimized to obtain the initial face generation model after model optimization, and based on the initial face generation model after model optimization, the target face generation model is determined, and the target face generation model is used to construct a three-dimensional face image.

[0010] According to another aspect of an embodiment of the present invention, a 3D face modeling device is provided, the device comprising:

[0011] An acquisition unit, configured to acquire a face training data set, wherein the face training data set includes a plurality of two-dimensional face training images and a training image label corresponding to each of the plurality of two-dimensional face training images;

[0012] A processing unit is used to call the encoder network in the initial face generation model to perform feature extraction on each of the two-dimensional face training images to obtain feature extraction results of the corresponding two-dimensional face training images; and determine the vector to be generated of the corresponding two-dimensional face training image based on the feature extraction results of each of the two-dimensional face training images;

[0013] The processing unit is further configured to call a generator in the initial face generation model to generate a three-dimensional face image corresponding to the corresponding two-dimensional face training image based on the vectors to be generated of each of the two-dimensional face training images; and determine a reflectivity image of the three-dimensional face image corresponding to each of the two-dimensional face training images;

[0014] The processing unit is further used to calculate the generation model loss value of the initial face generation model based on the reflectivity image of the three-dimensional face image corresponding to each of the two-dimensional face training images and the corresponding training image label;

[0015] The processing unit is also used to optimize the model parameters in the initial face generation model in the direction of reducing the loss value of the generation model to obtain the initial face generation model after model optimization, and determine the target face generation model based on the initial face generation model after model optimization, and the target face generation model is used to construct a three-dimensional face image.

[0016] According to another aspect of an embodiment of the present invention, an electronic device is provided, comprising a processor and a memory storing a program, wherein the program comprises instructions, which, when executed by the processor, enable the processor to perform the above-mentioned method.

[0017] According to another aspect of an embodiment of the present invention, a non-transitory computer-readable storage medium storing computer instructions is provided, wherein the computer instructions are used to enable a computer to execute the above-mentioned method.

[0018] After acquiring a face training data set, the embodiment of the present invention can call the encoder network in the initial face generation model to extract features from each two-dimensional face training image to obtain feature extraction results of the corresponding two-dimensional face training image. The face training data set includes multiple two-dimensional face training images and training image labels corresponding to each of the multiple two-dimensional face training images; and based on the feature extraction results of each two-dimensional face training image, the vector to be generated of the corresponding two-dimensional face training image is determined. Further, the generator in the initial face generation model can be called to generate a three-dimensional face image corresponding to the corresponding two-dimensional face training image based on the vector to be generated of each two-dimensional face training image; and the reflectivity image of the three-dimensional face image corresponding to each two-dimensional face training image is determined; thereby, based on the reflectivity image of the three-dimensional face image corresponding to each two-dimensional face training image and the corresponding training image label, the generation model loss value of the initial face generation model is calculated. Based on this, the model parameters in the initial face generation model can be optimized in the direction of reducing the loss value of the generation model to obtain the initial face generation model after model optimization, and based on the initial face generation model after model optimization, the target face generation model is determined, and the target face generation model is used to construct a three-dimensional face image, thereby effectively improving the model performance of the target face generation model to improve the accuracy of the target face generation model for modeling three-dimensional face images. It can be seen that the embodiment of the present invention can construct any two-dimensional face image into a corresponding three-dimensional face image through the target face generation model, so as to realize convenient three-dimensional face modeling and effectively improve the accuracy of three-dimensional face modeling; in addition, the embodiment of the present invention does not require multiple cameras or special equipment support, which can effectively reduce the hardware cost and complexity of the system. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] Further details, features and advantages of the invention are disclosed in the following description of exemplary embodiments in conjunction with the accompanying drawings, in which:

[0020] Figure 1 A schematic flow chart of a three-dimensional face modeling method according to an exemplary embodiment of the present invention is shown;

[0021] Figure 2 A schematic flow chart of another three-dimensional face modeling method according to an exemplary embodiment of the present invention is shown;

[0022] Figure 3 A schematic block diagram of a three-dimensional face modeling device according to an exemplary embodiment of the present invention is shown;

[0023] Figure 4 A block diagram of an exemplary electronic device that can be used to implement an embodiment of the present invention is shown. DETAILED DESCRIPTION

[0024] Embodiments of the present invention will be described in more detail below with reference to the accompanying drawings. Although certain embodiments of the present invention are shown in the accompanying drawings, it should be understood that the present invention can be implemented in various forms and should not be construed as being limited to the embodiments described herein, which are instead provided for a more thorough and complete understanding of the present invention. It should be understood that the drawings and embodiments of the present invention are only for exemplary purposes and are not intended to limit the scope of protection of the present invention.

[0025] It should be understood that the various steps described in the method embodiments of the present invention may be performed in different orders and / or in parallel. In addition, the method embodiments may include additional steps and / or omit the steps shown. The scope of the present invention is not limited in this respect.

[0026] The term "including" and its variations used in this document are open inclusions, that is, "including but not limited to". The term "based on" means "based at least in part on". The term "one embodiment" means "at least one embodiment"; the term "another embodiment" means "at least one other embodiment"; the term "some embodiments" means "at least some embodiments". Relevant definitions of other terms will be given in the following description. It should be noted that the concepts of "first", "second", etc. mentioned in the present invention are only used to distinguish different devices, modules or units, and are not used to limit the order or interdependence of the functions performed by these devices, modules or units.

[0027] It should be noted that the modifications of "one" and "plurality" mentioned in the present invention are illustrative rather than restrictive, and those skilled in the art should understand that, unless otherwise clearly indicated in the context, it should be understood as "one or more".

[0028] The names of the messages or information exchanged between multiple devices in the embodiments of the present invention are only used for illustrative purposes, and are not used to limit the scope of these messages or information.

[0029] It should be noted that the execution subject of the three-dimensional face modeling method provided in the embodiment of the present invention may be one or more electronic devices, which is not limited in the embodiment of the present invention; wherein, the electronic device may be a terminal (i.e., a client) or a server, then when the execution subject includes multiple electronic devices, and the multiple electronic devices include at least one terminal and at least one server, the three-dimensional face modeling method provided in the embodiment of the present invention may be jointly executed by the terminal and the server. Accordingly, the terminal mentioned here may include but is not limited to: a smart phone, a tablet computer, a laptop computer, a desktop computer, an intelligent voice interaction device, and the like. The server mentioned here may be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN (Content Delivery Network), and basic cloud computing services such as big data and artificial intelligence platforms, and the like.

[0030] Based on the above description, an embodiment of the present invention proposes a 3D face modeling method, which can be executed by the electronic device (terminal or server) mentioned above; or, the 3D face modeling method can be executed by the terminal and the server together. For the sake of convenience, the following description will be given by taking the electronic device executing the 3D face modeling method as an example; Figure 1 As shown, the 3D face modeling method may include the following steps S101-S105:

[0031] S101, obtaining a face training data set, where the face training data set includes a plurality of two-dimensional face training images and a training image label corresponding to each of the plurality of two-dimensional face training images.

[0032] Among them, the face training data set may include multiple face training data, and one face training data may include a two-dimensional face training image and a training image label corresponding to the corresponding two-dimensional face training image, so that the face training data set may include multiple two-dimensional face training images and training image labels corresponding to each of the multiple two-dimensional face training images.

[0033] Optionally, the training image label corresponding to a two-dimensional face training image can be a three-dimensional face image label corresponding to the corresponding two-dimensional face training image (i.e., a real three-dimensional face image), or it can be a reflectance image of the three-dimensional face image label corresponding to the corresponding two-dimensional face training image (i.e., the reflectance image label corresponding to the corresponding two-dimensional face training image), etc.; the embodiments of the present invention are not limited to this.

[0034] In the embodiment of the present invention, the face training data set may be obtained in the following ways, but not limited to:

[0035] The first acquisition method: the face training data set may be stored in the storage space of the electronic device itself. In this case, the face training data set may be acquired from the storage space itself.

[0036] The second acquisition method: the electronic device can obtain a download link for the face training data set. In this case, the face training data set can be downloaded based on the download link for the face training data set to obtain the face training data set.

[0037] The third acquisition method: the electronic device can obtain an initial face training data set, which may include multiple two-dimensional face training images and three-dimensional face image labels corresponding to each two-dimensional face training image; when the training image label corresponding to a two-dimensional face training image is the reflectivity image of the three-dimensional face image label corresponding to the corresponding two-dimensional face training image, for any two-dimensional face training image among the multiple two-dimensional face training images, the electronic device can determine the training image label corresponding to any two-dimensional face training image based on the three-dimensional face image label corresponding to any two-dimensional face training image (that is, determine the reflectivity image of the three-dimensional face image label corresponding to any two-dimensional face training image, and use the reflectivity image of the three-dimensional face image label corresponding to any two-dimensional face training image as the training image label corresponding to any two-dimensional face training image); then after obtaining the training image labels corresponding to each two-dimensional face training image, the face training data set can be obtained, and so on. Among them, the method for determining the reflectivity image of the three-dimensional face image label corresponding to any two-dimensional face training image is the same as the method for determining the reflectivity image of the three-dimensional face image (also referred to as the three-dimensional face generated image) corresponding to any of the following two-dimensional face training images, and the embodiments of the present invention will not be repeated here.

[0038] S102, calling the encoder network in the initial face generation model, respectively extracting features from each two-dimensional face training image to obtain feature extraction results of the corresponding two-dimensional face training image; and determining the vector to be generated of the corresponding two-dimensional face training image based on the feature extraction results of each two-dimensional face training image.

[0039] Optionally, a face generation model may include a generator in a generative adversarial network (GAN); correspondingly, a discrimination (also referred to as a discriminant) model may include a discriminator in a generative adversarial network; optionally, a face generation model may also include an encoder network and / or a mapping network, an encoder network may be a convolutional neural network, a mapping network may be a fully connected neural network or an artificial neural network, etc., and the embodiment of the present invention does not limit the specific structure of the face generation model and the discrimination model. Optionally, a generative adversarial network may be a StyleGAN model (an improved generative adversarial network), or a StyleGAN2 model (another improved generative adversarial network), etc., and the embodiment of the present invention does not limit this; that is, the face generation model and discrimination model mentioned in the embodiment of the present invention may respectively include the generator and discriminator in the StyleGAN model, or may respectively include the generator and discriminator in the StyleGAN2 model, etc., and the embodiment of the present invention does not limit this.

[0040] Optionally, the model parameters in the initial face generation model may be randomly generated, or may be set according to experience or actual needs, which is not limited in the embodiment of the present invention.

[0041] In an embodiment of the present invention, for any two-dimensional face training image among a plurality of two-dimensional face training images, the electronic device may call the encoder network in the initial face generation model to perform feature extraction on any two-dimensional face training image to obtain a feature extraction result of any two-dimensional face training image; accordingly, based on the feature extraction result of any two-dimensional face training image, a vector to be generated for any two-dimensional face training image may be determined. Optionally, a feature extraction result may be a feature representation vector of the corresponding two-dimensional face training image.

[0042] Optionally, when determining the vector to be generated of any two-dimensional face training image based on the feature extraction result of any two-dimensional face training image, the electronic device may randomly generate a noise vector according to a preset normal distribution (such as a standard normal distribution), and the dimension of the noise vector may be the same as the dimension of the feature extraction result of any two-dimensional face training image. In this case, the sum of the noise vector and the feature extraction result of any two-dimensional face training image may be used as the intermediate vector of any two-dimensional face training image, and the mapping network in the initial face generation model may be called to map the intermediate vector of any two-dimensional face training image to the vector to be generated of any two-dimensional face training image, that is, the mapping network in the initial face generation model may be called to determine the vector to be generated of any two-dimensional face training image based on the intermediate vector of any two-dimensional face training image; or, the mapping network in the initial face generation model may be called to map the feature extraction result of any two-dimensional face training image to the vector to be generated of any two-dimensional face training image, and so on; the embodiments of the present invention are not limited to this. The vector to be generated may also be referred to as a latent vector, and each dimension of the vector to be generated may correspond to a direction or feature in the latent space; optionally, the dimension of the vector to be generated may be higher than the dimension of the feature extraction result. Optionally, the preset normal distribution may be set according to experience or according to actual needs, which is not limited in the embodiment of the present invention.

[0043] S103, calling the generator in the initial face generation model, generating a three-dimensional face image corresponding to the corresponding two-dimensional face training image based on the vector to be generated of each two-dimensional face training image; and determining the reflectivity image of the three-dimensional face image corresponding to each two-dimensional face training image.

[0044] In an embodiment of the present invention, for any two-dimensional face training image among a plurality of two-dimensional face training images, the electronic device may call a generator in an initial face generation model, and based on a vector to be generated of any two-dimensional face training image, generate a three-dimensional face image corresponding to any two-dimensional face training image, thereby determining a reflectivity image of a three-dimensional face image corresponding to any two-dimensional face training image. In other words, the vector to be generated of any two-dimensional face training image may be used in a generator in an initial face generation model to generate a three-dimensional face image.

[0045] Optionally, when determining the reflectivity image of the three-dimensional face image corresponding to each two-dimensional face training image, the electronic device may traverse the three-dimensional face image corresponding to each two-dimensional face training image in the plurality of two-dimensional face training images, and use the three-dimensional face image corresponding to the currently traversed two-dimensional face training image as the current three-dimensional face image. Then, based on the current three-dimensional face image, the texture mapping image of the current three-dimensional face image may be determined; optionally, the electronic device may determine the texture image of the current three-dimensional face image based on the current three-dimensional face image, and texture map the texture image of the current three-dimensional face image to obtain the texture mapping image of the current three-dimensional face image. Optionally, the electronic device may call a 3D morphological model (such as 3DMM (3D Morphable Face Model, a statistical model of face shape and appearance, i.e., a deformable face model)) to determine the texture image of the current three-dimensional face image (also referred to as a UV texture image (a texture image under UV coordinates)) based on the current three-dimensional face image. That is, the texture image of the current three-dimensional face image may be output through the 3D morphological model, thereby converting the coordinate points of the current three-dimensional face image in three-dimensional space (i.e., three-dimensional space coordinate points) into two-dimensional texture coordinates; wherein, UV coordinates may also be referred to as texture coordinates, which are a type of two-dimensional coordinates.

[0046] Optionally, the texture image of a three-dimensional face image may include the texture coordinate points of each three-dimensional space coordinate point in the corresponding three-dimensional face image projected on the texture coordinates, that is, one three-dimensional space coordinate point may correspond to one texture coordinate point; Optionally, the electronic device may establish a corresponding relationship between the current three-dimensional face image and the corresponding texture image through a 3D morphological model, and may use a centroid interpolation method to obtain the RGB (RGB color mode) value of each texture coordinate point in the texture image of the current three-dimensional face image, or use the RGB value of the three-dimensional space coordinate point corresponding to any texture coordinate point as the RGB value of any texture coordinate point, so as to determine the texture image of the current three-dimensional face image. Optionally, the electronic device may also output the surface normal corresponding to each texture coordinate point in the texture image of the current three-dimensional face image through a 3D morphological model, the surface normal corresponding to one texture coordinate point may be the surface normal of the three-dimensional space coordinate point corresponding to the corresponding texture coordinate point, or the surface normal corresponding to one texture coordinate point may be the weighted summation result between the surface normals of the first K three-dimensional space coordinate points closest to the three-dimensional space coordinate point corresponding to the corresponding texture coordinate point, K may be a positive integer, and so on; the embodiment of the present invention is not limited to this.

[0047] Optionally, when texture mapping is performed on the texture image of the current three-dimensional face image to obtain the texture mapping image of the current three-dimensional face image, texture mapping can be performed on the texture image of the current three-dimensional face image to obtain a texture mapping result, and the texture mapping result can be used as the texture mapping image of the current three-dimensional face image; or, texture correction can be performed on the texture mapping result to obtain a texture correction result, and the texture correction result can be used as the texture mapping image of the current three-dimensional face image, thereby obtaining a texture mapping image with unchanged posture, and so on; the embodiments of the present invention are not limited to this.

[0048] Optionally, the coordinate range of a texture image may be between 0 and 1, that is, the UV coordinates usually represent the position in the texture image in a normalized manner; when texture mapping is performed on the texture image of the current three-dimensional face image to obtain a texture mapping result, the texture mapping size corresponding to the current three-dimensional face image may be determined, and the texture mapping size corresponding to the current three-dimensional face image may be texture mapped on the texture image of the current three-dimensional face image to obtain a texture mapping result, at which time the size of the texture mapping result may be the texture mapping size corresponding to the current three-dimensional face image, thereby being converted from UV coordinates to pixel coordinates, and the texture mapping result may be an image in pixel coordinates; optionally, the texture mapping size corresponding to the current three-dimensional face image may be a specified image size, or may be the size of the current three-dimensional face image in the horizontal direction (i.e., the size on the xy plane), etc., and the embodiment of the present invention does not limit this; optionally, the specified image size may be set according to experience or according to actual needs, and the embodiment of the present invention does not limit this.

[0049] Optionally, the electronic device may determine the mapping pixel coordinate points corresponding to each texture coordinate point in the texture image of the current three-dimensional face image (i.e., the texture coordinate points corresponding to each three-dimensional space coordinate point in the current three-dimensional face image) according to the texture mapping size corresponding to the current three-dimensional face image; for example, assuming that the texture mapping size corresponding to the current three-dimensional face image is 256×256, and any texture coordinate point is (0.23, 0.35), then the mapping pixel coordinate point corresponding to any texture coordinate point may be (58.88, 89.6). Based on this, the electronic device may determine the texture mapping result based on the mapping pixel coordinate points corresponding to each texture coordinate point and the texture image of the current three-dimensional face image; optionally, for any pixel point in the mapping texture result (i.e., any pixel point among multiple pixel points, the multiple pixel points may include each pixel point under the texture mapping size corresponding to the current three-dimensional face image), the electronic device may determine the first H mapping pixel coordinate points closest to any pixel point from the mapping pixel coordinate points corresponding to each texture coordinate point, and perform weighted summation on the RGB values ​​of the texture coordinate points corresponding to each of the determined H mapping pixel coordinate points. (For example, the weight corresponding to a mapping pixel coordinate may be negatively correlated with the distance between any pixel points, or a mean operation may be performed, etc.), to obtain the pixel value of any pixel point, where H is a positive integer; or, a set of mapping pixel coordinate points whose distance from any pixel point is less than a preset pixel distance threshold may be determined from the mapping pixel coordinate points corresponding to each texture coordinate point, and the RGB values ​​of the texture coordinate points corresponding to each mapping pixel point in the mapping pixel coordinate point set are weighted summed to obtain the pixel value of any pixel point, and so on; after obtaining the pixel values ​​of each pixel point in the texture mapping result, the texture mapping result is obtained to determine the texture mapping result. Optionally, the preset pixel distance threshold may be set according to experience or according to actual needs, and the embodiment of the present invention is not limited to this. Optionally, the electronic device may also determine the surface normal corresponding to each pixel point in the texture mapping result; accordingly, for any pixel point among the multiple pixel points, the surface normals corresponding to the texture coordinate points corresponding to each mapping pixel coordinate point in the determined H mapping pixel coordinate points may be weighted summed to obtain the surface normal corresponding to any pixel point; or, the surface normals corresponding to the texture coordinate points corresponding to each mapping pixel coordinate point in the mapping pixel coordinate point set may be weighted summed to obtain the surface normal corresponding to any pixel point, and so on; the embodiments of the present invention are not limited to this.

[0050] Optionally, when performing texture correction on the texture mapping result to obtain the texture correction result, the electronic device may perform image smoothing on the texture mapping result; and / or, may perform flat field correction on the texture mapping result; and / or, may perform Gamma correction on the texture mapping result to obtain the texture correction result, and so on; the embodiments of the present invention are not limited to this.

[0051] Further, the electronic device may determine the shadow image of the current three-dimensional face image based on the texture mapping image of the current three-dimensional face image; optionally, the electronic device may use a spherical harmonic lighting (SH, also known as spherical harmonic lighting) model to calculate the shadow pixel value of each pixel in the texture mapping image of the current three-dimensional face image (i.e., the shadow pixel value of each pixel among multiple pixels) based on the texture mapping image of the current three-dimensional face image, and use the shadow pixel value of any pixel as the pixel value of any pixel in the shadow image of the current three-dimensional face image (i.e., as the pixel value of any pixel in the shadow image of the current three-dimensional face image) to determine the shadow image of the current three-dimensional face image. Optionally, taking any pixel point as a pixel point of the i-th row and the j-th column as an example for explanation, i is less than or equal to the maximum number of rows in the texture mapping image of the current three-dimensional face image (i.e., the maximum number of rows indicated by the texture mapping size corresponding to the current three-dimensional face image), j is less than or equal to the maximum number of columns in the texture mapping image of the current three-dimensional face image (i.e., the maximum number of columns indicated by the texture mapping size corresponding to the current three-dimensional face image), and i and j are both positive integers; then the electronic device can use formula 1.1 to calculate the shadow pixel value of any pixel point:

[0052]

[0053] Among them, S ij It can represent the shadow pixel value of any pixel (i.e. the pixel at the i-th row and j-th column), L b It can represent the spherical harmonic coefficient of the bth spherical harmonic basis function, b can represent the index of the spherical harmonic basis function, H b It can represent the bth spherical harmonic basis function in the spherical harmonic illumination model, N ij It can represent the surface normal corresponding to any pixel point. Optionally, the spherical harmonic coefficients of a spherical harmonic basis function can be set according to experience or according to actual needs, which is not limited in the embodiment of the present invention.

[0054] Based on this, the electronic device can determine the reflectivity image of the current three-dimensional face image based on the texture mapping image and shadow image of the current three-dimensional face image; optionally, for any pixel point among multiple pixels, the pixel value of any pixel point in the reflectivity image of the current three-dimensional face image (also referred to as the reflectivity pixel value of any pixel point) can be calculated based on the pixel value of any pixel point in the texture mapping image of the current three-dimensional face image and the pixel value of any pixel point in the shadow image of the current three-dimensional face image (i.e., the shadow pixel value of any pixel point), so as to determine the reflectivity image of the current three-dimensional face image. Optionally, taking any pixel point as a pixel point in the i-th row and j-th column as an example for explanation; the electronic device can use formula 1.2 to calculate the reflectivity pixel value of any pixel point:

[0055] T ij =max(min(A ij ,δ),∈)⊙max(S )j ,∈) Formula 1.2

[0056] Among them, T ij It can represent the pixel value of any pixel in the texture mapping image of the current 3D face image. ij It can represent the reflectivity pixel value of any pixel point, so the reflectivity pixel value of any pixel point can be inferred by formula 1.2. It can be seen that the embodiment of the present invention can multiply the corresponding elements of the reflectivity image and the shadow image of the current three-dimensional face image to obtain a texture mapping image of the same dimension. Optionally, δ and ∈ can be set according to experience or according to actual needs, and the embodiment of the present invention is not limited to this.

[0057] Correspondingly, after traversing the three-dimensional face images corresponding to each of the multiple two-dimensional face training images, the reflectivity images of the three-dimensional face images corresponding to each of the two-dimensional face training images can be obtained to determine the reflectivity images of the three-dimensional face images corresponding to each of the two-dimensional face training images.

[0058] S104, calculating a generation model loss value of the initial face generation model based on the reflectance image of the three-dimensional face image corresponding to each two-dimensional face training image and the corresponding training image label.

[0059] In an embodiment of the present invention, for any two-dimensional face training image among multiple two-dimensional face training images, the electronic device may determine a reflectance image feature vector corresponding to any two-dimensional face training image based on the reflectance image of the three-dimensional face image corresponding to any two-dimensional face training image (a reflectance image feature vector is a feature vector extracted from a reflectance image, that is, the reflectance image feature vector of the reflectance image of the three-dimensional face image corresponding to any two-dimensional face training image may be used as the reflectance image feature vector corresponding to any two-dimensional face training image); and determine a reflectance image feature vector label corresponding to any two-dimensional face training image based on the training image label corresponding to any two-dimensional face training image. Optionally, when determining the reflectance image feature vector label corresponding to any two-dimensional face training image based on the training image label corresponding to any two-dimensional face training image, if the training image label corresponding to any two-dimensional face training image is a three-dimensional face image label, then the reflectance image of the three-dimensional face image label corresponding to any two-dimensional face training image (i.e., the reflectance image label corresponding to any two-dimensional face training image) can be determined based on the three-dimensional face image label corresponding to any two-dimensional face training image, and the reflectance image feature vector label corresponding to any two-dimensional face training image can be determined based on the reflectance image of the three-dimensional face image label corresponding to any two-dimensional face training image; if the training image label corresponding to any two-dimensional face training image is a reflectance image label, then the reflectance image feature vector label corresponding to any two-dimensional face training image can be determined based on the reflectance image label corresponding to any two-dimensional face training image (i.e., the reflectance image feature vector of the reflectance image label corresponding to any two-dimensional face training image can be used as the reflectance image feature vector label corresponding to any two-dimensional face training image), and so on. Optionally, when determining the reflectivity image feature vector of any reflectivity image, the intermediate layer network of the identification model (such as the first identification model) under the current iteration can be called to extract features of any reflectivity image to obtain the reflectivity image feature vector of any reflectivity image; optionally, the intermediate layer network of an identification model may include one or more layers of networks in the corresponding identification model, which is not limited in the embodiment of the present invention. It can be seen that the embodiment of the present invention can learn to distinguish the reflectivity image label corresponding to any two-dimensional face training image and the reflectivity image of the three-dimensional face image corresponding to any two-dimensional face training image through the identification model, and feed back to the face generation model in the form of the face generation model loss value to optimize the generation process.

[0060] Based on this, the electronic device can calculate the generation loss value under any two-dimensional face training image based on the reflectance image feature vector and the reflectance image feature vector label corresponding to any two-dimensional face training image; optionally, the electronic device can use formula 1.3 to calculate the generation loss value under any two-dimensional face training image:

[0061]

[0062] Among them, A r A can represent the true reflectance image (i.e., reflectance image label) corresponding to any two-dimensional face training image. g It can represent the reflectivity image of the 3D face image corresponding to any 2D face training image, D(A r ,A g ) can represent the generation loss value under any two-dimensional face training image; correspondingly, F(A r ) can represent the reflectivity image feature vector label corresponding to any two-dimensional face training image (i.e., the feature vector extracted from the reflectivity image label corresponding to any two-dimensional face training image), F(A g ) can represent the reflectivity image feature vector corresponding to any two-dimensional face training image (i.e., the feature vector extracted from the reflectivity image of the three-dimensional face image corresponding to any two-dimensional face training image). In an embodiment of the present invention, formula 1.3 can represent the cosine distance between the real reflectivity image and the reflectivity image of the three-dimensional face image generated by the model, and the stability of the model during the training process can be guaranteed by the cosine distance; wherein, the value range of the generation loss value under any two-dimensional face training image is [0, 2], when the generation loss value under any two-dimensional face training image is 0, it means that the reflectivity image of the three-dimensional face image generated by the face generation model has the highest similarity with the real reflectivity image, and when the generation loss value under any two-dimensional face training image is 2, it means that the reflectivity image of the three-dimensional face image generated by the face generation model is completely dissimilar to the real reflectivity image; based on this, the reflectivity image of the generated three-dimensional face image can be made consistent with the identity of the real face, and the original features of the face can be maintained.

[0063] Furthermore, after obtaining the generation loss value under each two-dimensional face training image, the generation loss value under each two-dimensional face training image can be weighted summed (such as summation operation or mean operation, etc.) to obtain the generation model loss value of the initial face generation model.

[0064] S105, optimizing the model parameters in the initial face generation model in the direction of reducing the loss value of the generation model, obtaining the initial face generation model after model optimization, and determining the target face generation model based on the initial face generation model after model optimization, and the target face generation model is used to construct a three-dimensional face image.

[0065] Exemplarily, the t-th model training (i.e., iterative training) of the face generation model is taken as an example for explanation; for any model parameter θ in the face generation model, the electronic device can use formula 1.4 to update any model parameter:

[0066]

[0067] Among them, α can represent the learning rate, L loss It can represent the loss value of the face generation model in the current iteration (i.e., the t-1th iteration) (for example, when the value of t is 1, it can be the model loss value of the initial face generation model), θ t It can represent the value of any model parameter at the tth iteration, θ t-1 It can represent the value of any model parameter at the t-1th iteration (for example, when the value of t is 1, it can be the value of any model parameter in the initial face generation model). It can represent the gradient of the loss value of the face generation model with respect to any model parameter.

[0068] In an embodiment of the present invention, by training the face generation model, the reflectance image of the three-dimensional face image generated by the face generation model can be made closer to the real reflectance image, thereby improving the authenticity and quality of the rendered image; that is, by making the symmetry and similarity of the generated image consistent with the real image, random artifacts or inconsistent parts in the rendered image can be reduced, thereby improving the quality and realism of the rendered image; that is, the embodiment of the present invention can enable the generated reflectance image to maintain consistency with the identity of the real face, that is, the generated image can be as close to the real image as possible, maintaining the original characteristics of the face.

[0069] After acquiring a face training data set, the embodiment of the present invention can call the encoder network in the initial face generation model to extract features from each two-dimensional face training image to obtain feature extraction results of the corresponding two-dimensional face training image. The face training data set includes multiple two-dimensional face training images and training image labels corresponding to each of the multiple two-dimensional face training images; and based on the feature extraction results of each two-dimensional face training image, the vector to be generated of the corresponding two-dimensional face training image is determined. Further, the generator in the initial face generation model can be called to generate a three-dimensional face image corresponding to the corresponding two-dimensional face training image based on the vector to be generated of each two-dimensional face training image; and the reflectivity image of the three-dimensional face image corresponding to each two-dimensional face training image is determined; thereby, based on the reflectivity image of the three-dimensional face image corresponding to each two-dimensional face training image and the corresponding training image label, the generation model loss value of the initial face generation model is calculated. Based on this, the model parameters in the initial face generation model can be optimized in the direction of reducing the loss value of the generation model to obtain the initial face generation model after model optimization, and based on the initial face generation model after model optimization, the target face generation model is determined, and the target face generation model is used to construct a three-dimensional face image, thereby effectively improving the model performance of the target face generation model to improve the accuracy of the target face generation model for modeling three-dimensional face images. It can be seen that the embodiment of the present invention can construct any two-dimensional face image into a corresponding three-dimensional face image through the target face generation model, so as to realize convenient three-dimensional face modeling and effectively improve the accuracy of three-dimensional face modeling; in addition, the embodiment of the present invention does not require multiple cameras or special equipment support, which can effectively reduce the hardware cost and complexity of the system.

[0070] Based on the above description, the embodiment of the present invention also proposes a more specific 3D face modeling method. Accordingly, the 3D face modeling method can be executed by the electronic device (terminal or server) mentioned above; or, the 3D face modeling method can be executed by the terminal and the server together. For the sake of convenience, the following description will be given by taking the electronic device executing the 3D face modeling method as an example; please refer to Figure 2 The 3D face modeling method may include the following steps S201-S209:

[0071] S201, obtaining a face training data set, where the face training data set includes a plurality of two-dimensional face training images and a training image label corresponding to each of the plurality of two-dimensional face training images.

[0072] S202, calling the encoder network in the initial face generation model, performing feature extraction on each two-dimensional face training image, respectively, to obtain feature extraction results of the corresponding two-dimensional face training image; and determining the vector to be generated of the corresponding two-dimensional face training image based on the feature extraction results of each two-dimensional face training image.

[0073] S203, calling the generator in the initial face generation model, generating a three-dimensional face image corresponding to the corresponding two-dimensional face training image based on the vector to be generated of each two-dimensional face training image; and determining the reflectivity image of the three-dimensional face image corresponding to each two-dimensional face training image.

[0074] S204, calculating a generation model loss value of the initial face generation model based on the reflectance image of the three-dimensional face image corresponding to each two-dimensional face training image and the corresponding training image label.

[0075] S205, optimizing the model parameters in the initial face generation model in a direction of reducing the loss value of the generation model, and obtaining the initial face generation model after model optimization.

[0076] S206, based on the initial face generation model after model optimization, determine the first intermediate face generation model; and call the first intermediate face generation model to generate first three-dimensional face images corresponding to each two-dimensional face training image, so as to determine the reflectivity image of the first three-dimensional face image corresponding to each two-dimensional face training image.

[0077] Optionally, the electronic device may use the initial face generation model after model optimization as the first intermediate face generation model, then after each face generation model training process (i.e., performing model training on the face generation model once), a recognition model training process (i.e., performing model training on the recognition model once); or, the initial face generation model after model optimization may continue to be iteratively trained until the number of iterations (i.e., the number of model trainings starting from the initial face generation model) is equal to W, and the face generation model after the Wth iteration (i.e., the Wth model training) is used as the first intermediate face generation model, where W is an integer greater than 1, then after each W face generation model training process (i.e., performing model training on the face generation model W times), V recognition model training processes (i.e., performing model training on the recognition model V times), where V is a positive integer, and so on; the embodiments of the present invention are not limited to this.

[0078] It should be noted that the method for determining the reflectivity image of any first three-dimensional face image may be as shown above, and the embodiment of the present invention will not be repeated here.

[0079] S207, based on the reflectivity image of the first three-dimensional face image corresponding to each two-dimensional face training image and the real sample data set, construct an identification training data set, the identification training data set includes multiple identification training samples, one identification training sample is a real sample or a generated sample, and the multiple identification training samples include M generated samples and N real samples, and M and N are both positive integers.

[0080] Optionally, the identification training data set may further include a sample label of each identification training sample in the plurality of identification training samples, wherein a sample label is a real sample label or a generated sample label, wherein a real sample label may be used to indicate that the corresponding identification training sample is a real sample, and a generated sample label may be used to indicate that the corresponding identification training sample is a generated sample. Optionally, both the real sample label and the generated sample label may be set according to experience or according to actual needs, which is not limited in the embodiment of the present invention.

[0081] Optionally, the electronic device may also obtain a real sample data set, the real sample data set may include N real sample data, one real sample data may include one real sample, that is, the real sample data set may include N real samples, N is a positive integer; optionally, one real sample data may also include a real sample label of the corresponding real sample, etc.; the embodiment of the present invention is not limited to this. Among them, a real sample can be a reflectance image of a real three-dimensional face image.

[0082] Optionally, the electronic device may obtain a real sample data set based on the training image labels corresponding to each two-dimensional face training image in the face training data set; illustratively, when the training image label corresponding to a two-dimensional face training image is a reflectance image label, N reflectance image labels may be selected from the face training data set, and the selected N reflectance image labels are all used as real samples to be added to the real sample data set; or, when the training image label corresponding to a two-dimensional face training image is a three-dimensional face image label, N three-dimensional face image labels may be selected from the face training data set, and the reflectance image of each three-dimensional face image label in the selected N three-dimensional face image labels is determined, so that the reflectance image of each selected three-dimensional face image label is used as a real sample to be added to the real sample data set, etc. Optionally, the electronic device may also obtain a real sample data download link, and use the reflectance images downloaded based on the real sample data download link as real samples to be added to the real sample data set, thereby achieving the acquisition of the real sample data set, etc.

[0083] Based on this, when constructing an identification training data set based on the reflectance image of the first three-dimensional face image corresponding to each two-dimensional face training image and the real sample data set, the electronic device can select the reflectance image of the first three-dimensional face image corresponding to each two-dimensional face training image in the M two-dimensional face training images based on the reflectance image of the first three-dimensional face image corresponding to each two-dimensional face training image, and use the reflectance image of the first three-dimensional face image corresponding to each two-dimensional face training image in the selected M two-dimensional face training images as a generated sample, thereby obtaining M generated samples, and adding the M generated samples to the identification training data set, thereby using each of the M generated samples as an identification training sample; optionally, when the identification training data set also includes sample labels of each identification training sample, the generated sample label can also be used as the sample label of each of the M generated samples, and M is a positive integer. Accordingly, each real sample in the real sample data set can be added to the identification training data set, so that each real sample in the real sample data set serves as an identification training sample; optionally, when the identification training data set also includes sample labels of each identification training sample, the real sample label can also be used as the sample label of each real sample to realize the construction of the identification training data set, so that multiple identification training samples may include M generated samples and N real samples, and so on.

[0084] S208, calling the first identification model to predict the true sample prediction probability of each identification training sample in the multiple identification training samples, respectively, where a true sample prediction probability is used to indicate the probability that the corresponding identification training sample is a true sample; and based on the true sample prediction probability of each identification training sample, calculating the identification model loss value of the first identification model.

[0085] In an embodiment of the present invention, the electronic device may calculate a real sample loss value (such as a binary cross entropy loss) based on the real sample prediction probability of each real sample in the identification training data set; correspondingly, it may calculate a generated sample loss value (such as a binary cross entropy loss) based on the real sample prediction probability of each generated sample in the identification training data set; based on this, a weighted summation (such as a mean operation or a summation operation, etc.) may be performed on the corresponding real sample loss values ​​and the generated sample loss values ​​to obtain the identification model loss value of the first identification model.

[0086] For example, the electronic device may use formula 2.1 to calculate the real sample loss value:

[0087]

[0088] Among them, L real It can represent the real sample loss value, N can be the number of real samples, A rncan be the nth real sample (i.e., the reflectivity image of the nth real 3D face image), D real It can represent the prediction probability of the identification model for the real sample of the identification training sample.

[0089] Accordingly, the electronic device can use formula 2.2 to calculate the sample loss value:

[0090]

[0091] Among them, L fake It can represent the loss value of generated samples, M can be the number of generated samples, A gm A sample may be generated for the mth time (ie, a reflectance image of the mth generated three-dimensional face image).

[0092] Based on this, the electronic device can use formula 2.3 to calculate the identification model loss value of the first identification model:

[0093] L D =L real +L fake Formula 2.3

[0094] Among them, L D The discriminant model loss value of the first discriminant model may be represented.

[0095] S209, optimize the model parameters in the first identification model in the direction of reducing the loss value of the identification model to obtain the first identification model after model optimization, and after determining the second identification model based on the first identification model after model optimization, determine the target face generation model based on the first intermediate face generation model, and the target face generation model is used to construct a three-dimensional face image.

[0096] Optionally, for any model parameter θ in the discriminant model D , the electronic device can use formula 2.4 to update any model parameter θ D :

[0097]

[0098] Among them, η can represent the learning rate, It can be expressed as the loss value of the identification model relative to any model parameter θ D Optionally, a gradient can be calculated by a back propagation algorithm, or by a numerical method, etc.; this is not limited in the embodiment of the present invention.

[0099] Optionally, the first identification model can be an initial identification model, in which case the electronic device can first perform model training on the face generation model, and then perform model training on the identification model; or, the first identification model can also be an identification model after the initial identification model has been trained once or V times, in which case the identification model can be trained first, and then the face generation model can be trained, and so on; the embodiment of the present invention is not limited to this. Optionally, the model parameters in the initial identification model can be randomly generated, or can be set according to experience or actual needs, and the embodiment of the present invention is not limited to this. It should be understood that each time a face generation model training process is performed, that is, a identification model training process is performed, the second identification model can be the first identification model after model optimization; or, each time a face generation model training process is performed W times, that is, a identification model training process is performed V times, the second identification model can be an identification model after the first identification model is trained V times, and so on.

[0100] Furthermore, when determining the target face generation model based on the first intermediate face generation model, the electronic device may continue to iteratively train the first intermediate face generation model; after obtaining the second intermediate face generation model, continue to iteratively train the second identification model through the second intermediate face generation model (that is, call the second intermediate face generation model to generate three-dimensional face images of each two-dimensional face training image in the current iteration, so as to construct the identification training data set in the current iteration, and continue to iteratively train the second identification model), so as to alternately implement the face generation model training process and the identification model training process, until the true sample prediction probability of each identification training sample in the current iteration meets the convergence condition, so as to obtain the target face generation model, that is, when the true sample prediction probability of each identification training sample in the current iteration meets the convergence condition, the model training of the face generation model and the identification model can be stopped, and the face generation model after stopping the model training is used as the target face generation model.

[0101] Optionally, if the real sample prediction probability of each identification training sample under the current iteration is equal to the target probability value, it can be determined that the real sample prediction probability of each identification training sample under the current iteration meets the convergence condition; or, if the difference between the real sample prediction probability of each identification training sample under the current iteration and the target probability value is less than the first preset probability difference, it can be determined that the real sample prediction probability of each identification training sample under the current iteration meets the convergence condition; or, based on the real sample prediction probability of each identification training sample under the current iteration, the number of fuzzy identification samples can be determined. If the ratio of the number of fuzzy identification samples to the total number of identification training samples in the identification training data set is greater than the preset ratio threshold, it is determined that the real sample prediction probability of each identification training sample under the current iteration meets the convergence condition. The number of fuzzy identification samples can refer to the number of identification training samples whose difference between the real sample prediction probability and the target probability value is less than the second preset probability difference, etc. Optionally, the target probability value, the first preset probability difference, the second preset probability difference and the preset ratio threshold can be set according to experience or according to actual needs, and the embodiment of the present invention does not limit this. Exemplarily, the target probability value can be 0.5.

[0102] It should be understood that as the iterations proceed, the face generation model will continuously adjust its parameters to make the generated 3D face images closer to the real 3D face images, thereby reducing the loss of the identification model. At the same time, the identification model will also continuously adjust its parameters to improve its ability to distinguish between real samples and generated samples, thereby increasing the loss of the identification model. Ultimately, the losses of the face generation model and the identification model will converge to a stable state, at which point the face generation model can generate high-quality 3D face images.

[0103] It can be seen that the embodiment of the present invention uses an alternating descent optimization method, which can achieve coordination and consistency between the three-dimensional face texture and shape, and can effectively reduce the complexity of calculation. Exemplarily, the three-dimensional face texture (i.e., reflectivity image) can be fixed, and the three-dimensional face shape model can be optimized to make the rendered three-dimensional face image as close as possible to the input two-dimensional face image. At this stage, the goal of the face generation model can be to generate a three-dimensional shape as close to the real face as possible. Correspondingly, the three-dimensional face shape model can be fixed, and the three-dimensional face texture can be optimized to match the identity and appearance of the generated three-dimensional face texture with the input two-dimensional image. At this stage, the goal of the identification model can be to judge the difference between the generated three-dimensional face texture and the real image. These two stages are performed alternately until convergence is reached. Based on this, the embodiment of the present invention realizes the training of high-quality shapes and high-quality facial information without supervision, so that high-quality three-dimensional face images can be generated from two-dimensional face images. Furthermore, the embodiments of the present invention can generate a corresponding three-dimensional face texture from the vector to be generated of the two-dimensional face image (i.e., generate a corresponding three-dimensional face image and determine the reflectance image of the corresponding three-dimensional face image), and can learn to generate a reflectance image with invariant illumination and posture through self-supervision while maintaining identity consistency and realism, which can effectively improve the model performance of the face generation model and the identification model, and thereby improve the accuracy of three-dimensional face modeling.

[0104] Further, after obtaining the target face generation model, the electronic device can obtain the target two-dimensional face image, which can be any two-dimensional face image, and the embodiment of the present invention does not limit this; and call the encoder network in the target face generation model to extract features of the target two-dimensional face image to obtain the feature extraction result of the target two-dimensional face image; thereby determining the vector to be generated of the target two-dimensional face image based on the feature extraction result of the target two-dimensional face image. Optionally, the electronic device can call the mapping network in the target face generation model to determine the vector to be generated of the target two-dimensional face image based on the feature extraction result of the target two-dimensional face image, that is, the feature extraction result of the target two-dimensional face image can be mapped to the vector to be generated of the target two-dimensional face image through the mapping network in the target face generation model. Then correspondingly, the electronic device can call the generator in the target face generation model to generate a three-dimensional face image corresponding to the target two-dimensional face image based on the vector to be generated of the target two-dimensional face image. Based on this, the embodiments of the present invention can obtain a high-quality three-dimensional face image corresponding to a target two-dimensional face image through a target face generation model with higher model performance, thereby effectively improving the accuracy of three-dimensional face modeling, thereby effectively improving the accuracy of the generated three-dimensional face image.

[0105] After acquiring a face training data set, the embodiment of the present invention can call the encoder network in the initial face generation model to extract features from each two-dimensional face training image to obtain feature extraction results of the corresponding two-dimensional face training image. The face training data set includes multiple two-dimensional face training images and training image labels corresponding to each of the multiple two-dimensional face training images; and based on the feature extraction results of each two-dimensional face training image, the vector to be generated of the corresponding two-dimensional face training image is determined. Then, the generator in the initial face generation model can be called to generate a three-dimensional face image corresponding to the corresponding two-dimensional face training image based on the vector to be generated of each two-dimensional face training image; and the reflectivity image of the three-dimensional face image corresponding to each two-dimensional face training image is determined; thereby, based on the reflectivity image of the three-dimensional face image corresponding to each two-dimensional face training image and the corresponding training image label, the generation model loss value of the initial face generation model is calculated. Based on this, the model parameters in the initial face generation model can be optimized in the direction of reducing the loss value of the generation model to obtain the initial face generation model after model optimization; thus, based on the initial face generation model after model optimization, the first intermediate face generation model is determined; and the first intermediate face generation model is called to generate the first three-dimensional face image corresponding to each two-dimensional face training image respectively, so as to determine the reflectivity image of the first three-dimensional face image corresponding to each two-dimensional face training image. Further, based on the reflectivity image of the first three-dimensional face image corresponding to each two-dimensional face training image and the real sample data set, a recognition training data set can be constructed, the recognition training data set includes multiple recognition training samples, one recognition training sample is a real sample or a generated sample, and the multiple recognition training samples include M generated samples and N real samples, and M and N are both positive integers; the first recognition model is called to predict the real sample prediction probability of each recognition training sample respectively, and a real sample prediction probability is used to indicate the probability that the corresponding recognition training sample is a real sample; and based on the real sample prediction probability of each recognition training sample, the recognition model loss value of the first recognition model is calculated. Based on this, the model parameters in the first identification model can be optimized in the direction of reducing the loss value of the identification model to obtain the first identification model after model optimization. After the second identification model is determined based on the first identification model after model optimization, the target face generation model is determined based on the first intermediate face generation model. The target face generation model is used to construct a three-dimensional face image, thereby continuously improving the model performance of the face generation model and the identification model in the model training process (which may include the face generation model training process and the identification model training process) to obtain a target face generation model with higher model performance.It can be seen that the embodiment of the present invention can construct any two-dimensional face image into a corresponding three-dimensional face image through the target face generation model, so as to realize convenient three-dimensional face modeling and effectively improve the accuracy of three-dimensional face modeling; in addition, the embodiment of the present invention does not require multiple cameras or special equipment support, which can effectively reduce the hardware cost and complexity of the system.

[0106] Based on the description of the above-mentioned related embodiments of the 3D face modeling method, the embodiment of the present invention further proposes a 3D face modeling device, which can be a computer program (including program code) running in an electronic device; Figure 3 As shown, the 3D face modeling device may include an acquisition unit 301 and a processing unit 302. The 3D face modeling device may execute Figure 1 or Figure 2 The three-dimensional face modeling method shown, that is, the three-dimensional face modeling device can run the above units:

[0107] An acquisition unit 301 is used to acquire a face training data set, wherein the face training data set includes a plurality of two-dimensional face training images and a training image label corresponding to each of the plurality of two-dimensional face training images;

[0108] The processing unit 302 is used to call the encoder network in the initial face generation model to perform feature extraction on each of the two-dimensional face training images to obtain feature extraction results of the corresponding two-dimensional face training images; and determine the vector to be generated of the corresponding two-dimensional face training image based on the feature extraction results of each of the two-dimensional face training images;

[0109] The processing unit 302 is further configured to call a generator in the initial face generation model to generate a three-dimensional face image corresponding to the corresponding two-dimensional face training image based on the vectors to be generated of each two-dimensional face training image; and determine a reflectivity image of the three-dimensional face image corresponding to each two-dimensional face training image;

[0110] The processing unit 302 is further configured to calculate a generation model loss value of the initial face generation model based on the reflectance image of the three-dimensional face image corresponding to each of the two-dimensional face training images and the corresponding training image label;

[0111] The processing unit 302 is also used to optimize the model parameters in the initial face generation model in the direction of reducing the loss value of the generation model to obtain the initial face generation model after model optimization, and determine the target face generation model based on the initial face generation model after model optimization, and the target face generation model is used to construct a three-dimensional face image.

[0112] In one implementation, when determining the reflectance image of the three-dimensional face image corresponding to each of the two-dimensional face training images, the processing unit 302 may be specifically configured to:

[0113] Traversing the three-dimensional face images corresponding to each of the multiple two-dimensional face training images, and taking the three-dimensional face image corresponding to the currently traversed two-dimensional face training image as the current three-dimensional face image;

[0114] Based on the current three-dimensional face image, determining a texture mapping image of the current three-dimensional face image; and based on the texture mapping image of the current three-dimensional face image, determining a shadow image of the current three-dimensional face image;

[0115] Determining a reflectivity image of the current three-dimensional face image based on the texture mapping image and the shadow image of the current three-dimensional face image;

[0116] After traversing the three-dimensional face images corresponding to each of the multiple two-dimensional face training images, the reflectivity images of the three-dimensional face images corresponding to each of the two-dimensional face training images are obtained to determine the reflectivity images of the three-dimensional face images corresponding to each of the two-dimensional face training images.

[0117] In another implementation manner, when the processing unit 302 determines the texture mapping image of the current three-dimensional face image based on the current three-dimensional face image, it can be specifically used to:

[0118] Based on the current three-dimensional face image, determining a texture image of the current three-dimensional face image;

[0119] Texture mapping is performed on the texture image of the current three-dimensional face image to obtain a texture mapping image of the current three-dimensional face image.

[0120] In another implementation manner, when the processing unit 302 determines the target face generation model based on the initial face generation model after model optimization, it can be specifically used to:

[0121] Based on the initial face generation model optimized by the model, a first intermediate face generation model is determined; and the first intermediate face generation model is called to respectively generate first three-dimensional face images corresponding to the two-dimensional face training images, so as to determine reflectivity images of the first three-dimensional face images corresponding to the two-dimensional face training images;

[0122] Based on the reflectivity image of the first three-dimensional face image corresponding to each of the two-dimensional face training images and the real sample data set, constructing a discrimination training data set, the discrimination training data set comprising a plurality of discrimination training samples, one discrimination training sample being a real sample or a generated sample, the plurality of discrimination training samples comprising M generated samples and N real samples, M and N being positive integers;

[0123] Calling the first identification model to predict the true sample prediction probability of each identification training sample in the multiple identification training samples, respectively, where a true sample prediction probability is used to indicate the probability that the corresponding identification training sample is a true sample; and calculating the identification model loss value of the first identification model based on the true sample prediction probability of each identification training sample;

[0124] In the direction of reducing the loss value of the identification model, the model parameters in the first identification model are optimized to obtain the first identification model after model optimization, and after the second identification model is determined based on the first identification model after model optimization, the target face generation model is determined based on the first intermediate face generation model.

[0125] In another implementation manner, when the processing unit 302 determines the target face generation model based on the first intermediate face generation model, it can be specifically used to:

[0126] Continuing to iteratively train the first intermediate face generation model;

[0127] After obtaining the second intermediate face generation model, the second identification model is continued to be iteratively trained through the second intermediate face generation model to alternately implement the face generation model training process and the identification model training process until the true sample prediction probability of each identification training sample under the current iteration meets the convergence condition, thereby obtaining the target face generation model.

[0128] In another implementation, when the processing unit 302 calculates the generation model loss value of the initial face generation model based on the reflectance image of the three-dimensional face image corresponding to each of the two-dimensional face training images and the corresponding training image label, it can be specifically used to:

[0129] For any two-dimensional face training image among the plurality of two-dimensional face training images, based on a reflectivity image of a three-dimensional face image corresponding to the any two-dimensional face training image, determining a reflectivity image feature vector corresponding to the any two-dimensional face training image;

[0130] Determine a reflectivity image feature vector label corresponding to any two-dimensional face training image based on a training image label corresponding to any two-dimensional face training image;

[0131] Calculate the generation loss value of any two-dimensional face training image based on the reflectance image feature vector and the reflectance image feature vector label corresponding to any two-dimensional face training image;

[0132] After obtaining the generation loss values ​​under the respective two-dimensional face training images, weighted summation is performed on the generation loss values ​​under the respective two-dimensional face training images to obtain the generation model loss value of the initial face generation model.

[0133] In another implementation, the acquisition unit 301 may also be used to:

[0134] Obtain a target two-dimensional face image;

[0135] The processing unit 302 may also be used to:

[0136] Calling the encoder network in the target face generation model to perform feature extraction on the target two-dimensional face image to obtain a feature extraction result of the target two-dimensional face image; and determining a vector to be generated of the target two-dimensional face image based on the feature extraction result of the target two-dimensional face image;

[0137] The generator in the target face generation model is called to generate a three-dimensional face image corresponding to the target two-dimensional face image based on the vector to be generated of the target two-dimensional face image.

[0138] According to one embodiment of the present invention, Figure 1 or Figure 2 Each step involved in the method shown can be Figure 3 The three-dimensional face modeling device shown in the figure is executed by each unit. For example, Figure 1 The step S101 shown in FIG. 1 can be performed by Figure 3 The acquisition unit 301 shown in FIG. 1 is executed, and steps S102-S105 can be performed by Figure 3 The processing unit 302 shown in is executed. For example, Figure 2 The step S201 shown in FIG. 1 can be performed by Figure 3 The acquisition unit 301 shown in FIG. 1 is executed, and steps S202-S209 can be performed by Figure 3 The processing unit 302 shown in executes, and so on.

[0139] According to another embodiment of the present invention, Figure 3Each unit in the three-dimensional face modeling device shown can be separately or completely combined into one or several other units to form, or one (some) of the units can be further divided into multiple functionally smaller units to form, which can achieve the same operation without affecting the realization of the technical effects of the embodiments of the present invention. The above-mentioned units are divided based on logical functions. In practical applications, the function of one unit can also be realized by multiple units, or the functions of multiple units can be realized by one unit. In other embodiments of the present invention, any three-dimensional face modeling device can also include other units. In practical applications, these functions can also be implemented with the assistance of other units, and can be implemented by the collaboration of multiple units.

[0140] According to another embodiment of the present invention, the program can be executed by running a program on a general electronic device such as a computer including a central processing unit (CPU), a random access memory medium (RAM), a read-only memory medium (ROM), and other processing elements and storage elements. Figure 1 or Figure 2 A computer program (including program code) for each step involved in the corresponding method shown in Figure 3 The three-dimensional face modeling device shown in and the three-dimensional face modeling method of the embodiment of the present invention are implemented. The computer program can be recorded on a computer storage medium, for example, and loaded into the above-mentioned electronic device through the computer storage medium and run therein.

[0141] After acquiring a face training data set, the embodiment of the present invention can call the encoder network in the initial face generation model to extract features from each two-dimensional face training image to obtain feature extraction results of the corresponding two-dimensional face training image. The face training data set includes multiple two-dimensional face training images and training image labels corresponding to each of the multiple two-dimensional face training images; and based on the feature extraction results of each two-dimensional face training image, the vector to be generated of the corresponding two-dimensional face training image is determined. Further, the generator in the initial face generation model can be called to generate a three-dimensional face image corresponding to the corresponding two-dimensional face training image based on the vector to be generated of each two-dimensional face training image; and the reflectivity image of the three-dimensional face image corresponding to each two-dimensional face training image is determined; thereby, based on the reflectivity image of the three-dimensional face image corresponding to each two-dimensional face training image and the corresponding training image label, the generation model loss value of the initial face generation model is calculated. Based on this, the model parameters in the initial face generation model can be optimized in the direction of reducing the loss value of the generation model to obtain the initial face generation model after model optimization, and based on the initial face generation model after model optimization, the target face generation model is determined, and the target face generation model is used to construct a three-dimensional face image, thereby effectively improving the model performance of the target face generation model to improve the accuracy of the target face generation model for modeling three-dimensional face images. It can be seen that the embodiment of the present invention can construct any two-dimensional face image into a corresponding three-dimensional face image through the target face generation model, so as to realize convenient three-dimensional face modeling and effectively improve the accuracy of three-dimensional face modeling; in addition, the embodiment of the present invention does not require multiple cameras or special equipment support, which can effectively reduce the hardware cost and complexity of the system.

[0142] Based on the description of the above method embodiment and device embodiment, the exemplary embodiment of the present invention further provides an electronic device, including: at least one processor; and a memory connected to the at least one processor in communication. The memory stores a computer program that can be executed by the at least one processor, and the computer program is used to enable the electronic device to perform the method according to the embodiment of the present invention when executed by the at least one processor.

[0143] Exemplary embodiments of the present invention also provide a non-transitory computer-readable storage medium storing a computer program, wherein the computer program, when executed by a processor of a computer, is used to cause the computer to perform a method according to an embodiment of the present invention.

[0144] An exemplary embodiment of the present invention further provides a computer program product, comprising a computer program, wherein when the computer program is executed by a processor of a computer, the computer is used to enable the computer to perform a method according to an embodiment of the present invention.

[0145] refer to Figure 4 , a block diagram of an electronic device 400 that can be used as a server or client of the present invention will now be described, which is an example of a hardware device that can be applied to various aspects of the present invention. The electronic device is intended to represent various forms of digital electronic computer devices, such as laptop computers, desktop computers, workbenches, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present invention described and / or required herein.

[0146] like Figure 4 As shown, the electronic device 400 includes a computing unit 401, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 402 or a computer program loaded from a storage unit 408 into a random access memory (RAM) 403. In the RAM 403, various programs and data required for the operation of the electronic device 400 can also be stored. The computing unit 401, the ROM 402, and the RAM 403 are connected to each other via a bus 404. An input / output (I / O) interface 405 is also connected to the bus 404.

[0147] A plurality of components in the electronic device 400 are connected to the I / O interface 405, including: an input unit 406, an output unit 407, a storage unit 408, and a communication unit 409. The input unit 406 may be any type of device capable of inputting information to the electronic device 400, and the input unit 406 may receive input digital or character information, and generate key signal inputs related to user settings and / or function control of the electronic device. The output unit 407 may be any type of device capable of presenting information, and may include, but is not limited to, a display, a speaker, a video / audio output terminal, a vibrator, and / or a printer. The storage unit 408 may include, but is not limited to, a disk, an optical disk. The communication unit 409 allows the electronic device 400 to exchange information / data with other devices via a computer network such as the Internet and / or various telecommunication networks, and may include, but is not limited to, a modem, a network card, an infrared communication device, a wireless communication transceiver, and / or a chipset, such as a Bluetooth™ device, a WiFi device, a WiMax device, a cellular communication device, and / or the like.

[0148] The computing unit 401 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the computing unit 401 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microcontrollers, etc. The computing unit 401 performs the various methods and processes described above. For example, in some embodiments, the three-dimensional face modeling method may be implemented as a computer software program, which is tangibly contained in a machine-readable medium, such as a storage unit 408. In some embodiments, part or all of the computer program may be loaded and / or installed on the electronic device 400 via the ROM 402 and / or the communication unit 409. In some embodiments, the computing unit 401 may be configured to perform the three-dimensional face modeling method in any other appropriate manner (e.g., by means of firmware).

[0149] The program code for implementing the method of the present invention can be written in any combination of one or more programming languages. These program codes can be provided to a processor or controller of a general-purpose computer, a special-purpose computer or other programmable data processing device, so that the program code, when executed by the processor or controller, enables the functions / operations specified in the flow chart and / or block diagram to be implemented. The program code can be executed entirely on the machine, partially on the machine, partially on the machine as a stand-alone software package and partially on a remote machine, or entirely on a remote machine or server.

[0150] In the context of the present invention, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices, or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0151] As used herein, the terms "machine-readable medium" and "computer-readable medium" refer to any computer program product, apparatus, and / or device (e.g., disk, optical disk, memory, programmable logic device (PLD)) for providing machine instructions and / or data to a programmable processor, including a machine-readable medium that receives machine instructions as a machine-readable signal. The term "machine-readable signal" refers to any signal for providing machine instructions and / or data to a programmable processor.

[0152] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0153] The systems and techniques described herein may be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), and the Internet.

[0154] A computer system may include clients and servers. Clients and servers are generally remote from each other and usually interact through a communication network. The relationship of client and server is generated by computer programs running on respective computers and having a client-server relationship to each other.

[0155] Furthermore, it should be understood that what is disclosed above is only a preferred embodiment of the present invention, and certainly cannot be used to limit the scope of rights of the present invention. Therefore, equivalent changes made according to the claims of the present invention are still within the scope of the present invention.

Claims

1. A three-dimensional face modeling method, characterized in that: include: Acquire a face training data set, wherein the face training data set includes a plurality of two-dimensional face training images and a training image label corresponding to each of the plurality of two-dimensional face training images; Calling the encoder network in the initial face generation model to perform feature extraction on each of the two-dimensional face training images to obtain feature extraction results of the corresponding two-dimensional face training images; and determining the vectors to be generated of the corresponding two-dimensional face training images based on the feature extraction results of the two-dimensional face training images respectively; Calling the generator in the initial face generation model to generate three-dimensional face images corresponding to the corresponding two-dimensional face training images based on the vectors to be generated of the two-dimensional face training images respectively; and determining the reflectivity images of the three-dimensional face images corresponding to the two-dimensional face training images; Calculating a generation model loss value of the initial face generation model based on the reflectance images of the three-dimensional face images corresponding to the two-dimensional face training images and the corresponding training image labels; In the direction of reducing the loss value of the generation model, the model parameters in the initial face generation model are optimized to obtain the initial face generation model after model optimization, and based on the initial face generation model after model optimization, the target face generation model is determined, and the target face generation model is used to construct a three-dimensional face image.

2. The method according to claim 1, characterized in that The step of determining the reflectivity image of the three-dimensional face image corresponding to each of the two-dimensional face training images comprises: Traversing the three-dimensional face images corresponding to each of the multiple two-dimensional face training images, and taking the three-dimensional face image corresponding to the currently traversed two-dimensional face training image as the current three-dimensional face image; Based on the current three-dimensional face image, determining a texture mapping image of the current three-dimensional face image; and based on the texture mapping image of the current three-dimensional face image, determining a shadow image of the current three-dimensional face image; Determining a reflectivity image of the current three-dimensional face image based on the texture mapping image and the shadow image of the current three-dimensional face image; After traversing the three-dimensional face images corresponding to each of the multiple two-dimensional face training images, the reflectivity images of the three-dimensional face images corresponding to each of the two-dimensional face training images are obtained to determine the reflectivity images of the three-dimensional face images corresponding to each of the two-dimensional face training images.

3. The method according to claim 2, characterized in that The step of determining a texture mapping image of the current three-dimensional face image based on the current three-dimensional face image comprises: Based on the current three-dimensional face image, determining a texture image of the current three-dimensional face image; Texture mapping is performed on the texture image of the current three-dimensional face image to obtain a texture mapping image of the current three-dimensional face image.

4. The method according to any one of claims 1 to 3, characterized in that: The step of determining a target face generation model based on the initial face generation model optimized by the model includes: Based on the initial face generation model optimized by the model, a first intermediate face generation model is determined; and the first intermediate face generation model is called to respectively generate first three-dimensional face images corresponding to the two-dimensional face training images, so as to determine reflectivity images of the first three-dimensional face images corresponding to the two-dimensional face training images; Based on the reflectivity image of the first three-dimensional face image corresponding to each of the two-dimensional face training images and the real sample data set, constructing a discrimination training data set, the discrimination training data set comprising a plurality of discrimination training samples, one discrimination training sample being a real sample or a generated sample, the plurality of discrimination training samples comprising M generated samples and N real samples, M and N being positive integers; Calling the first identification model to predict the true sample prediction probability of each identification training sample in the multiple identification training samples, respectively, where a true sample prediction probability is used to indicate the probability that the corresponding identification training sample is a true sample; and calculating the identification model loss value of the first identification model based on the true sample prediction probability of each identification training sample; In the direction of reducing the loss value of the identification model, the model parameters in the first identification model are optimized to obtain the first identification model after model optimization, and after the second identification model is determined based on the first identification model after model optimization, the target face generation model is determined based on the first intermediate face generation model.

5. The method according to claim 4, characterized in that The step of determining a target face generation model based on the first intermediate face generation model includes: Continuing to iteratively train the first intermediate face generation model; After obtaining the second intermediate face generation model, the second identification model is continued to be iteratively trained through the second intermediate face generation model to alternately implement the face generation model training process and the identification model training process until the true sample prediction probability of each identification training sample under the current iteration meets the convergence condition, thereby obtaining the target face generation model.

6. The method according to any one of claims 1 to 3, characterized in that: The calculating the generation model loss value of the initial face generation model based on the reflectivity image of the three-dimensional face image corresponding to each of the two-dimensional face training images and the corresponding training image label comprises: For any two-dimensional face training image among the plurality of two-dimensional face training images, based on a reflectivity image of a three-dimensional face image corresponding to the any two-dimensional face training image, determining a reflectivity image feature vector corresponding to the any two-dimensional face training image; Determine a reflectivity image feature vector label corresponding to any two-dimensional face training image based on a training image label corresponding to any two-dimensional face training image; Calculate the generation loss value of any two-dimensional face training image based on the reflectance image feature vector and the reflectance image feature vector label corresponding to any two-dimensional face training image; After obtaining the generation loss values ​​under the respective two-dimensional face training images, weighted summation is performed on the generation loss values ​​under the respective two-dimensional face training images to obtain the generation model loss value of the initial face generation model.

7. The method according to any one of claims 1 to 3, characterized in that: The method further comprises: Obtain a target two-dimensional face image; Calling the encoder network in the target face generation model to perform feature extraction on the target two-dimensional face image to obtain a feature extraction result of the target two-dimensional face image; and determining a vector to be generated of the target two-dimensional face image based on the feature extraction result of the target two-dimensional face image; The generator in the target face generation model is called to generate a three-dimensional face image corresponding to the target two-dimensional face image based on the vector to be generated of the target two-dimensional face image.

8. A three-dimensional face modeling device, characterized in that: The device comprises: An acquisition unit, configured to acquire a face training data set, wherein the face training data set includes a plurality of two-dimensional face training images and a training image label corresponding to each of the plurality of two-dimensional face training images; a processing unit, configured to call an encoder network in an initial face generation model, perform feature extraction on each of the two-dimensional face training images, obtain feature extraction results of the corresponding two-dimensional face training images; and determine a vector to be generated for the corresponding two-dimensional face training image based on the feature extraction results of each of the two-dimensional face training images; The processing unit is further configured to call a generator in the initial face generation model to generate a three-dimensional face image corresponding to the corresponding two-dimensional face training image based on the vectors to be generated of each two-dimensional face training image; and determine a reflectivity image of the three-dimensional face image corresponding to each two-dimensional face training image; The processing unit is further used to calculate the generation model loss value of the initial face generation model based on the reflectance image of the three-dimensional face image corresponding to each of the two-dimensional face training images and the corresponding training image label; The processing unit is also used to optimize the model parameters in the initial face generation model in the direction of reducing the loss value of the generation model to obtain the initial face generation model after model optimization, and determine the target face generation model based on the initial face generation model after model optimization, and the target face generation model is used to construct a three-dimensional face image.

9. An electronic device, characterized in that: include: processor; as well as Memory for storing programs, The program includes instructions, which, when executed by the processor, cause the processor to perform the method according to any one of claims 1 to 7.

10. A non-transitory computer-readable storage medium storing computer instructions, wherein: The computer instructions are used to make a computer execute the method according to any one of claims 1-7.

Images