Image generation, three-dimensional model generation method, device, electronic device and medium

Abstract

The present disclosure provides an image generation, a three-dimensional model generation method, an apparatus, an electronic device and a medium, wherein the facial image generation method comprises: obtaining a normal vector image and texture feature data; wherein the pixel value of each pixel point in the normal vector image represents the value of the normal vector of the model vertex corresponding to the pixel point in the three-dimensional facial model corresponding to the normal vector image; and performing multi-level data fusion processing on the normal vector image and the texture feature data to obtain a reconstructed facial image.

Description

Technical Field

[0001] The present disclosure relates to the field of image processing technology, and in particular to an image generation, a three-dimensional model generation method, a device, an electronic device and a medium. Background Art

[0002] 3D face reconstruction refers to restoring a 3D model of a face based on a face image. After generating the 3D model of the face, a face image can be generated based on the 3D model of the face.

[0003] The current face image generation method has the problem of poor accuracy of the generated face images. Summary of the invention

[0004] The embodiments of the present disclosure at least provide a method, device, electronic device and medium for generating an image and a three-dimensional model.

[0005] In a first aspect, an embodiment of the present disclosure provides a method for generating a facial image, comprising: obtaining a normal vector image and texture feature data; wherein the pixel value of each pixel point in the normal vector image represents the value of the normal vector of the model vertex corresponding to the pixel point in a three-dimensional facial model corresponding to the normal vector image; and performing multi-level data fusion processing on the normal vector image and the texture feature data to obtain a reconstructed facial image.

[0006] In this way, the normal vector image is used as a geometric condition, and the texture feature data is used to model other influencing factors on the face image. By fusing the normal vector image and the texture feature data, the reconstructed face image has a higher accuracy.

[0007] In one possible implementation, the multi-level data fusion processing is performed on the normal vector image and the texture feature data to obtain a reconstructed facial image, including: performing feature transformation on the texture feature data to obtain transformed texture feature data corresponding to the multi-level data fusion processing; for each level of data fusion processing in the multi-level data fusion processing, performing a first feature fusion on the transformed texture feature data corresponding to the data fusion processing at that level and the input feature map corresponding to the data fusion processing at that level to obtain an intermediate feature map of the data fusion processing at that level; performing a second feature fusion on the intermediate feature map of the data fusion processing at that level and the normal vector image to obtain a result feature map corresponding to the data fusion processing at that level; and obtaining the reconstructed facial image based on the result feature map corresponding to the last level of data fusion processing.

[0008] In this way, the decoupling and fusion between the normal vector image and the texture feature data is achieved, thereby improving the realism and accuracy of the reconstructed facial image.

[0009] In a possible implementation, the feature transformation of the texture feature data to obtain transformed texture feature data corresponding to the multi-level data fusion processing includes: performing a first full connection processing on the texture feature data to obtain first intermediate texture feature data; performing multiple second full connection processing on the first intermediate texture feature data to obtain transformed texture feature data corresponding to the multi-level data fusion processing; wherein the full connection parameters of different second full connection processing are different.

[0010] In this way, the texture feature data is feature transformed, so that the transformed texture feature data corresponding to the multi-level data fusion processing formed after the transformation can not only improve the adaptability with the target three-dimensional face model, but also ensure that the transformed texture feature data corresponding to the data fusion processing at different levels include different features in the texture feature data, and then for each level of data fusion processing, the transformed texture feature data and the normal vector image corresponding to each level of data fusion processing are fused, thereby realizing the gradual fusion of the texture feature data and the normal vector image, so that the reconstructed face image obtained has higher accuracy and stronger sense of reality.

[0011] In a possible implementation, for each level of data fusion processing in the multi-level data fusion processing, before performing a first feature fusion on the transformed texture feature data corresponding to the level of data fusion processing and the input feature map corresponding to the level of data fusion processing to obtain the intermediate feature map of the level of data fusion processing, the method also includes: for the case where the level of data fusion processing is the first level of data fusion processing, upsampling the preset feature map to obtain the input feature map corresponding to the level of data fusion processing; for the case where the level of data fusion processing is not the first level of data fusion processing, upsampling the result feature map output by the previous level of data fusion processing of the level of data fusion processing to obtain the input feature map corresponding to the level of data fusion processing.

[0012] In this way, based on the preset feature map, the result feature map of each level of data fusion processing is upsampled step by step, so that the size of the result feature map of the last level of data fusion processing can match the size of the generated reconstructed face image, so that the generated face image has a higher resolution, thereby improving the clarity of the reconstructed face image.

[0013] In a possible implementation, for each level of data fusion processing in the multi-level data fusion processing, a first feature fusion is performed on the transformed texture feature data corresponding to the data fusion processing at that level and the input feature map corresponding to the data fusion processing at that level to obtain an intermediate feature map of the data fusion processing at that level, including: for each level of data fusion processing in the multi-level data fusion processing, using the transformed texture feature data corresponding to the data fusion processing at that level, a convolution kernel corresponding to the fusion processing at that level is transformed to obtain a transformed convolution kernel; and the input feature map is convolved with the transformed convolution kernel to obtain an intermediate feature map of the data fusion processing at that level.

[0014] Using this method, since the convolution kernel is transformed by transforming the texture feature data, and the data amount of the convolution kernel is usually smaller than the data amount of the input feature map, it can greatly reduce the data that needs to be processed during the transformation process and improve data processing efficiency.

[0015] In a possible implementation, for each level of data fusion processing in the multi-level data fusion processing, a first feature fusion is performed on the transformed texture feature data corresponding to the data fusion processing at that level and the input feature map corresponding to the data fusion processing at that level to obtain an intermediate feature map of the data fusion processing at that level, including: for each level of data fusion processing in the multi-level data fusion processing, using the transformed texture feature data corresponding to the data fusion processing at that level, the input feature map corresponding to the fusion processing at that level is transformed to obtain a transformed feature map; using the convolution kernel corresponding to the data fusion processing at that level, convolution processing is performed on the transformed feature map to obtain an intermediate feature map corresponding to the data fusion processing at that level.

[0016] In one possible implementation, the intermediate feature map of the data fusion processing at this level and the normal vector image are subjected to a second feature fusion to obtain a result feature map corresponding to the data fusion processing at this level, including: determining a product result matrix of the intermediate feature map of the data fusion processing at this level and the normal vector image; and determining the result feature map corresponding to the data fusion processing at this level based on the product result matrix.

[0017] In a possible implementation, determining the result feature graph corresponding to the data fusion processing at this level based on the product result matrix includes: obtaining the result feature graph corresponding to the data fusion processing at this level based on the product result matrix corresponding to the data fusion processing at this level, and a preset deviation matrix and / or noise matrix for the data fusion processing at this level.

[0018] Here, since the texture feature data of the face cannot fully express all the details of the face image, in order to make the finally generated reconstructed face image more realistic, after obtaining the product result matrix, based on the product result matrix corresponding to the data fusion processing at this level, and the preset deviation matrix and / or noise matrix of the data fusion processing at this level, the result feature map corresponding to the data fusion processing at this level can be obtained, so that the generated reconstructed face image can include the detailed features of the face and be more realistic.

[0019] In a possible implementation, the result feature map corresponding to the data fusion processing at this level is obtained based on the product result matrix corresponding to the data fusion processing at this level, and the preset deviation matrix and / or noise matrix of the data fusion processing at this level, including: performing positional addition on the product result matrix corresponding to the data fusion processing at this level, and the preset deviation matrix and / or noise matrix of the data fusion processing at this level, to obtain the result feature map corresponding to the data fusion processing at this level.

[0020] In a possible implementation manner, the facial image generation method is implemented by a pre-trained first target neural network.

[0021] In a possible implementation, the first target neural network is obtained in the following manner: a first sample normal vector image and first sample texture feature data of a first sample three-dimensional face model are obtained; using the first neural network to be trained, the first sample normal vector image and the first sample texture feature data are subjected to data fusion processing to obtain a reconstructed image of the first sample three-dimensional face model; based on the reconstructed image, a first training loss is obtained, and the first training loss is used to train the first neural network to be trained to obtain the first target neural network.

[0022] In a possible implementation, the first training loss includes at least one of the following: normal vector consistency loss, facial key point consistency loss, identity consistency loss, and adversarial loss.

[0023] In one possible implementation, the first training loss includes a normal vector consistency loss, and obtaining the first training loss based on the reconstructed image includes: performing normal vector prediction processing on the reconstructed image to obtain a predicted normal vector image of the reconstructed image; and using the first sample normal vector image and the predicted normal vector image to obtain the normal vector consistency loss.

[0024] In a possible implementation, the first training loss includes a key point consistency loss, and the first sample texture feature data includes: first sample texture feature data and second sample texture feature data; the reconstructed image includes a first reconstructed image obtained based on the first sample texture feature data, and a second reconstructed image obtained based on the second sample texture feature data; obtaining the first training loss based on the reconstructed image includes: performing key point recognition on the first reconstructed image to obtain a first key point of the first reconstructed image; and performing key point recognition based on the second reconstructed image to obtain a second key point of the second reconstructed image; and obtaining the key point consistency loss using the first key point and the second key point.

[0025] In a possible implementation manner, the first training loss includes identity consistency loss, and the first sample normal vector image of the first sample three-dimensional face model includes: a first sample normal vector image, and a second sample normal vector image; wherein the facial expressions and / or facial postures corresponding to the first sample normal vector image and the second sample normal vector image are different; the first neural network to be trained is used to perform data fusion processing on the first sample normal vector image and the first sample texture feature data to obtain a reconstructed image generated by reconstructing the first sample three-dimensional face model, including: using the first neural network to be trained, the first sample normal vector image and the first sample texture feature data are used to perform data fusion processing to obtain a third reconstructed image of the first sample three-dimensional face model; using the first neural network to be trained, the second sample normal vector image and the first sample texture feature data are used to perform data fusion processing to obtain a fourth reconstructed image of the first sample three-dimensional face model; based on the reconstructed image, obtaining the first training loss includes: performing identity recognition on the third reconstructed image to obtain a first identity recognition result; and performing identity recognition on the fourth reconstructed image to obtain a second identity recognition result; based on the first identity recognition result and the second identity recognition result, obtaining the identity consistency loss.

[0026] In a second aspect, the embodiments of the present disclosure also provide a method for generating a three-dimensional face model, comprising: performing three-dimensional face reconstruction and texture feature recovery on a target face image to obtain an initial normal vector image of the target face and initial texture feature data of the target face; based on the initial normal vector image and the initial texture feature data, obtaining a three-dimensional face model of the target face.

[0027] In this way, the initial normal vector image of the target face is predicted based on the target face image, and then the three-dimensional face model of the target face is obtained based on the initial normal vector image and the initial texture feature data. In this method, since the face image generated based on the initial normal vector image and the initial texture feature number has higher accuracy, the generated three-dimensional face model of the target face also has higher precision.

[0028] In a possible implementation, obtaining the three-dimensional face model of the target face based on the initial normal vector image and the initial texture feature data includes: using the initial normal vector image as the current normal vector image and the initial texture feature data as the current texture feature data, and generating a current reconstructed face image based on the current normal vector image and the current texture feature data; adjusting the current normal vector based on the target face image and the current reconstructed face image to obtain a target normal vector image of the target face; and generating a three-dimensional face model of the target face based on the target normal vector image.

[0029] In one possible implementation, the generating of a reconstructed facial image based on the current normal vector image and the texture feature data includes: using the current normal vector image as the normal vector image and the current texture feature data as the texture feature data, and utilizing the facial image generation method described in any one of the first aspects to generate a reconstructed facial image; and using the reconstructed facial image as the current reconstructed facial image generated based on the current normal vector image and the current reconstructed facial image.

[0030] In a possible implementation, the current normal vector is adjusted based on the target face image and the current reconstructed face image to obtain a target normal vector image of the target face, including: executing the following iterative process until the second loss is less than a preset loss threshold: obtaining a second loss based on the target face image and the current reconstructed face image; using the second loss to adjust the current normal vector image and the current texture feature data to obtain a new normal vector image and new texture feature data; using the new normal vector image as the current normal vector image and the new texture feature data as the current texture feature data, and returning to the step of generating the current reconstructed face image based on the current normal vector image and the current texture feature data; using the current normal vector image in the last iteration as the target normal vector image.

[0031] In this way, through multiple rounds of iterations, the face image generation method provided by the embodiment of the present disclosure can generate a three-dimensional face model that gradually approaches the target face.

[0032] In a possible implementation, the second loss includes: pixel consistency loss and / or classification feature consistency loss.

[0033] In one possible implementation, the second loss includes a classification feature consistency loss, and obtaining the second loss based on the target face image and the current reconstructed face image includes: using a pre-trained image classification network to classify the target face image to obtain first feature data output by a target network layer of the image classification network; and using the image classification network to classify the current reconstructed face image to obtain second feature data output by the target network layer; obtaining the classification feature consistency loss based on the first feature data and the second feature data; wherein the image classification network is trained using the target face image and the current reconstructed face image.

[0034] In one possible implementation, texture feature restoration is performed on a target facial image including a target face to obtain initial texture feature data of the target face, including: using a pre-trained second target neural network to perform texture feature restoration on the target facial image to obtain initial texture feature data of the target face; wherein the second target neural network is trained using the first target neural network described in the first aspect.

[0035] In one possible implementation, the second target neural network is trained in the following manner: using the first target neural network, the second sample normal vector image and the second sample texture feature data of the sample three-dimensional face model are processed to obtain a sample face image of the sample three-dimensional face model; using the second neural network to be trained to process the sample face image to obtain predicted texture feature data corresponding to the sample face image; based on the predicted texture feature data and the second sample texture feature data, a third loss is determined; based on the third loss, the second neural network to be trained is trained to obtain a second target neural network.

[0036] In a third aspect, an embodiment of the present disclosure provides a facial image generation device, comprising:

[0037] The first acquisition module is used to acquire a normal vector image and texture feature data; wherein the pixel value of each pixel point in the normal vector image represents the value of the normal vector of the model vertex corresponding to the pixel point in the three-dimensional face model corresponding to the normal vector image; the first processing module is used to perform multi-level data fusion processing on the normal vector image and the texture feature data to obtain a reconstructed face image.

[0038] In a fourth aspect, an embodiment of the present disclosure also provides a three-dimensional face model generating device, comprising: a second acquisition module, used to perform three-dimensional face reconstruction and texture feature recovery on a target face image, to obtain an initial normal vector image of the target face, and initial texture feature data of the target face; a second processing module, used to obtain a three-dimensional face model of the target face based on the initial normal vector image and the initial texture feature data.

[0039] In a fifth aspect, an optional implementation of the present disclosure further provides an electronic device, a processor, and a memory, wherein the memory stores machine-readable instructions executable by the processor, and the processor is used to execute the machine-readable instructions stored in the memory, and when the machine-readable instructions are executed by the processor, the machine-readable instructions perform the steps in the above-mentioned first aspect, or any possible implementation of the first aspect;

[0040] Or execute the steps in the above-mentioned second aspect, or any possible implementation manner of the second aspect.

[0041] In a sixth aspect, an optional implementation of the present disclosure further provides a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed, the steps of the first aspect or any possible implementation of the first aspect are executed;

[0042] Or execute the steps in the above-mentioned second aspect, or any possible implementation manner of the second aspect.

[0043] In order to make the above-mentioned objectives, features and advantages of the present disclosure more obvious and easy to understand, preferred embodiments are specifically cited below and described in detail with reference to the attached drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] In order to more clearly illustrate the technical solutions of the embodiments of the present disclosure, the following is a brief introduction to the drawings required for use in the embodiments. The drawings herein are incorporated into the specification and constitute a part of the specification. These drawings illustrate embodiments consistent with the present disclosure and are used together with the specification to illustrate the technical solutions of the present disclosure. It should be understood that the following drawings only illustrate certain embodiments of the present disclosure and should not be regarded as limiting the scope. For ordinary technicians in this field, other relevant drawings can also be obtained based on these drawings without creative work.

[0045] Figure 1 A flow chart of a method for generating a face image provided by an embodiment of the present disclosure is shown;

[0046] Figure 2A flowchart showing a specific method for performing multi-level data fusion processing on a normal vector image and texture feature data provided by an embodiment of the present disclosure is shown;

[0047] Figure 3 A flowchart showing a specific method for training a first target neural network provided by an embodiment of the present disclosure;

[0048] Figure 4 A schematic diagram of the structure of a first target neural network provided by an embodiment of the present disclosure is shown;

[0049] Figure 5 A schematic diagram of the structure of a rendering block in a first target neural network provided by an embodiment of the present disclosure is shown;

[0050] Figure 6 A flowchart of a method for generating a three-dimensional face model provided by an embodiment of the present disclosure is shown;

[0051] Figure 7 A schematic diagram of a facial image generating device provided by an embodiment of the present disclosure is shown;

[0052] Figure 8 A schematic diagram of a three-dimensional face model generating device provided by an embodiment of the present disclosure is shown;

[0053] Fig. 9 A schematic diagram of an electronic device provided by an embodiment of the present disclosure is shown. DETAILED DESCRIPTION

[0054] In order to make the purpose, technical scheme and advantages of the embodiments of the present disclosure clearer, the technical scheme in the embodiments of the present disclosure will be clearly and completely described below in conjunction with the drawings in the embodiments of the present disclosure. Obviously, the described embodiments are only part of the embodiments of the present disclosure, rather than all of the embodiments. The components of the embodiments of the present disclosure generally described and shown here can be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of the present disclosure is not intended to limit the scope of the present disclosure claimed for protection, but merely represents the selected embodiments of the present disclosure. Based on the embodiments of the present disclosure, all other embodiments obtained by those skilled in the art without making creative work belong to the scope of protection of the present disclosure.

[0055] Research has found that using a monocular face image as input, 3D face reconstruction of the face can restore the 3D face model. The 3D face model includes: multiple model vertices, and the connection relationship between the model vertices; the model vertices are connected to each other to form multiple face patches, which can represent the contour of the face. The purpose of 3D geometric reconstruction of the 3D face is to restore the 3D position information of each model vertex in the multiple model vertices that constitute the 3D face model. The current method of generating a 3D face model based on deep learning technology is usually to take the face image as input and learn to regress the corresponding 3D face model parameters, such as using a 3D deformable model (3DMM) to reconstruct the image into 3D face. However, this method of 3D face reconstruction requires a large amount of labeled data, which is often difficult to obtain, and the accuracy of the generated 3D face model is low.

[0056] In addition, generating facial images based on three-dimensional facial models plays an important role in the fields of film, television, games, virtual social networking, etc. For example, after the three-dimensional facial model of the original face is generated, the model vertices in the three-dimensional facial model can be adjusted in a certain way to generate a new facial image. In this way, the expression of the original face can be changed, so that the generated facial image has both the characteristics of the original face and the characteristics of a certain style. The current method of generating facial images using three-dimensional facial models usually uses an optimization method. This method obtains facial parameters such as reflectivity, texture, illumination, and viewing angle, and then generates facial images based on the obtained facial parameters according to certain graphic rules; however, the graphic rules usually use simplified models to describe the physical process of facial image capture, and many details of the imaging process cannot be modeled, resulting in poor accuracy of the generated facial images.

[0057] Based on the above research, the present disclosure provides a method for generating facial images, which uses the normal vector image of the face as a geometric condition and uses texture feature data to model other influencing factors on the facial image. By performing data fusion processing on the normal vector image and texture feature data, the reconstructed facial image obtained has higher accuracy.

[0058] In addition, the embodiments of the present disclosure also provide a method for generating a three-dimensional face model, by predicting an initial normal vector image of the target face based on a target face image of the target face, and then obtaining a three-dimensional face model of the target face based on the initial normal vector image and the initial texture feature data. In this method, since the face image based on the initial normal vector image and the initial texture feature number has a higher accuracy, the generated three-dimensional face model of the target face also has a higher precision.

[0059] The defects existing in the above solutions are the results obtained by the inventor after practice and careful research. Therefore, the discovery process of the above problems and the solutions proposed by the present disclosure for the above problems below should be the contributions made by the inventor to the present disclosure during the disclosure process.

[0060] It should be noted that similar reference numerals and letters denote similar items in the following drawings, and therefore, once an item is defined in one drawing, further definition and explanation thereof is not required in subsequent drawings.

[0061] To facilitate understanding of the present embodiment, a method for generating a facial image disclosed in the embodiment of the present disclosure is first introduced in detail. The execution subject of the facial image generation provided in the embodiment of the present disclosure is generally an electronic device with a certain computing capability, and the electronic device includes, for example: a terminal device or a server or other processing device, and the terminal device can be a user equipment (User Equipment, UE), a mobile device, a user terminal, a terminal, a cellular phone, a cordless phone, a personal digital assistant (Personal Digital Assistant, PDA), a handheld device, a computing device, a vehicle-mounted device, a wearable device, etc. In some possible implementations, the facial image generation can be implemented by a processor calling a computer-readable instruction stored in a memory.

[0062] The generation of a facial image provided by an embodiment of the present disclosure is described below.

[0063] See also Figure 1 FIG. 1 is a flowchart of a method for generating a facial image according to an embodiment of the present disclosure, wherein the method comprises steps S101 to S102, wherein:

[0064] S101: Acquire a normal vector image and texture feature data; wherein the pixel value of each pixel point in the normal vector image represents the value of the normal vector of the model vertex corresponding to the pixel point in the three-dimensional face model corresponding to the normal vector image.

[0065] S102: Perform multi-level data fusion processing on the normal vector image and the texture feature data to obtain a reconstructed face image.

[0066] The disclosed embodiment obtains a normal vector image and texture feature data of a three-dimensional face model, and performs multi-level data fusion processing on the normal vector image and the texture feature data to obtain a reconstructed face image, thereby using the normal vector image of the face as a geometric condition and using the texture feature data to model other influencing factors on the face image, and the resulting reconstructed face image has higher accuracy.

[0067] The above S101 to S102 are described in detail below.

[0068] Regarding S101 above:

[0069] The three-dimensional face model, for example, is obtained by performing three-dimensional face reconstruction on an image including a face, and the three-dimensional face model obtained by performing three-dimensional face reconstruction on an image including a face is directly determined as the three-dimensional face model for face image reconstruction; or, after performing three-dimensional face reconstruction on an image including a face to obtain a three-dimensional face model, the positions of each model vertex in the reconstructed three-dimensional face model in the three-dimensional coordinate system are adjusted to obtain the three-dimensional face model for face image reconstruction.

[0070] Here, the adjustment is, for example, adjustment for the fatness or thinness of the face, adjustment for facial expression, or adjustment based on a certain style.

[0071] The normal vector image of the three-dimensional face model can be generated, for example, using the normal vector of each model vertex in the three-dimensional face model.

[0072] In a specific implementation, in the three-dimensional coordinate system where the three-dimensional face model is located, the value of the normal vector of each model vertex includes the coordinate values ​​corresponding to each coordinate axis in the three-dimensional coordinate system. Then the normal vector image is, for example, a three-channel image, and the pixel value of each pixel in the image represents the coordinate values ​​of the normal vector of the model vertex corresponding to the pixel in the three coordinate axes.

[0073] When generating a normal vector image of a three-dimensional face model, for example, for each model vertex in the three-dimensional face model, at least one mesh with the model vertex as one of its vertices can be determined, and then the normal vector corresponding to at least one mesh can be determined according to the plane where the at least one mesh is located; and then the normal vector corresponding to the model vertex can be determined using the normal vector corresponding to the at least one mesh. When determining the normal vector corresponding to the model vertex using the normal vector corresponding to the at least one mesh, for example, the normal vector corresponding to the at least one mesh can be averaged to obtain the normal vector corresponding to the model vertex.

[0074] The texture feature data can be obtained by performing random Gaussian sampling on any image of a human face. Specifically, for different application scenarios of the embodiments of the present disclosure, the method of obtaining texture feature data is different; for example, if the application scenario is to generate a reconstructed face image with a certain style, random Gaussian sampling of texture information is performed on the face image of the style to obtain texture feature data; if a reconstructed face image under a certain lighting environment is to be generated, random Gaussian sampling of texture information is performed on the face image under the lighting environment to obtain texture feature data. Specifically, the extraction source of texture feature data is determined according to different needs.

[0075] Regarding S102 above:

[0076] See also Figure 2 As shown, the embodiment of the present disclosure provides a specific method for performing multi-level data fusion processing on the normal vector image and the texture feature data to obtain a reconstructed face image, including:

[0077] S201: Perform feature transformation on the texture feature data to obtain transformed texture feature data corresponding to the multi-level data fusion processing.

[0078] In a specific implementation, since the texture feature data obtained by random Gaussian sampling may be determined by using another image including a human face, if the human face image used to collect the texture feature data and the corresponding three-dimensional human face model cannot be adapted, if the texture feature data and the normal vector image are directly subjected to data fusion processing, the reconstructed human face image of the generated three-dimensional human face model may have a problem of poor authenticity. Therefore, in the embodiment of the present disclosure, when the normal vector image and the texture feature data are subjected to multi-level data fusion processing, the texture feature data will be subjected to feature transformation, so that the transformed texture feature data corresponding to the multi-level data fusion processing formed after the transformation can not only improve the adaptability with the three-dimensional human face model, but also ensure that the transformed texture feature data corresponding to the data fusion processing at different levels include different features in the texture feature data, and then for each level of data fusion processing, the transformed texture feature data corresponding to each level of data fusion processing and the normal vector image are fused, thereby realizing the gradual fusion of the texture feature data and the normal vector image, so that the obtained reconstructed human face image has higher accuracy and stronger sense of reality.

[0079] The present disclosure provides a specific method for performing feature transformation on texture feature data to obtain transformed texture feature data corresponding to multi-level data fusion processing, including:

[0080] The texture feature data is subjected to a first full connection process to obtain first intermediate texture feature data; the first intermediate texture feature data is subjected to multiple second full connection processes to obtain transformed texture feature data corresponding to the multi-level data fusion process; wherein the full connection parameters of different second full connection processes are different.

[0081] When performing the first fully connected processing on the texture feature data, for example, a first fully connected network having multiple fully connected layers can be used to perform the first fully connected processing on the texture feature data to obtain first intermediate texture feature data.

[0082] When the texture feature data is subjected to multiple second fully connected processing, for example, multiple second fully connected networks can be used to respectively perform fully connected processing on the first intermediate texture feature data to obtain transformed texture feature data corresponding to the multiple second fully connected networks respectively.

[0083] Among them, each second fully connected network corresponds to a level of data fusion processing, and different second fully connected networks have different network parameters. Different second fully connected networks can extract different texture features from the first intermediate texture feature data, thereby improving the authenticity of the generated reconstructed face image by dividing different texture features into multiple levels and gradually fusing them with the normal vector image.

[0084] Exemplarily, the texture feature data is represented by: z; after the first full connection processing is performed on the texture feature data, the first intermediate texture feature data obtained is represented by z'. Then the relationship between z and z' is represented by z'=M 1 (z); the transformed texture feature data is represented as w, and after the second full connection processing is performed on the first intermediate texture feature data, the relationship between z′ and w is represented as: w=M(z′). 1 (·) indicates the first full connection process; M 2 (·) represents the second full connection processing; then the texture feature data is subjected to feature transformation, and the transformed texture feature data corresponding to the multi-level data fusion processing can be simplified as the following formula (1):

[0085] w=M(z) (1)

[0086] Following the above S201, the method for performing multi-level data fusion processing on the normal vector image and texture feature data provided by the embodiment of the present disclosure also includes:

[0087] S202: For each level of data fusion processing in the multi-level data fusion processing, a first feature fusion is performed on the transformed texture feature data corresponding to the data fusion processing at that level and the input feature map corresponding to the data fusion processing at that level to obtain an intermediate feature map of the data fusion processing at that level.

[0088] In a specific implementation, for each level of data fusion processing, there is an input feature map corresponding to the data fusion processing at that level. Among them, for the case where the data fusion processing at that level is the first level of data fusion processing, the input feature map corresponding to the first level of data fusion processing is determined using a preset feature map. For different three-dimensional face models and texture feature data, the preset feature map can be the same. In the case of using an image generation neural network to implement the face image generation method provided in the example of the present disclosure, the preset feature map can be determined in the process of training the image generation neural network. The specific determination method can be referred to below Figure 3 The corresponding embodiments are shown and will not be described again here.

[0089] Here, for the first-level data fusion processing, the preset feature map can be directly determined as the input feature map corresponding to the first-level data fusion processing, or the preset feature map can be upsampled to obtain the input feature map corresponding to the first-level data fusion processing.

[0090] In addition, for each level of data fusion processing, if the data fusion processing at this level is not the first level of data fusion processing, the input feature map corresponding to the data fusion processing at this level is determined by using the result feature map output by the previous level of data fusion processing.

[0091] Here, for each level of data fusion processing except the first level of data fusion processing, the result feature map output by the corresponding previous level of data fusion processing can be directly used as the input feature map corresponding to the data fusion processing at this level; in addition, the result feature map output by the previous level of data fusion processing can also be upsampled to obtain the input feature map corresponding to the data fusion processing at this level.

[0092] In this way, based on the preset feature map, the result feature map of each level of data fusion processing is upsampled step by step, so that the size of the result feature map of the last level of data fusion processing can match the size of the generated reconstructed face image, so that the generated reconstructed face image has a higher resolution, thereby improving the clarity of the reconstructed face image.

[0093] Exemplarily, taking the result feature map output by the previous level of data fusion processing as an example, when the result feature map is upsampled, for example, each feature point in the result feature map (together with the feature value) is copied and filled to the adjacent position of the corresponding feature point. If the size of the result feature map is m*n, the size of the feature map obtained after copying and filling is 2m*2n; then, the feature map formed after copying and filling is Gaussian filtered to obtain the input feature map corresponding to the data fusion processing at this level.

[0094] The specific method of performing a first feature fusion on the transformed texture feature data corresponding to the data fusion processing of the level and the input feature map corresponding to the data fusion processing of the level to obtain the intermediate feature map of the data fusion processing of the level may be, for example, any one of the following A or B:

[0095] A:

[0096] For each level of data fusion processing in the multi-level data fusion processing, the convolution kernel corresponding to the level of data fusion processing is transformed using the transformed texture feature data corresponding to the level of data fusion processing to obtain the transformed convolution kernel.

[0097] For example, based on the following formula (2), the transformed texture feature data corresponding to the data fusion processing at this level can be used to transform the convolution kernel corresponding to the data fusion processing at this level:

[0098]

[0099] Among them, k cij is the convolution kernel parameter at position (i, j) in the cth convolution kernel channel corresponding to the data fusion processing at this level; w c represents the texture feature element corresponding to the cth convolution kernel channel in the transformed texture feature data; the number of channels of the convolution kernel, the number of channels of the input feature map, and the number of texture feature elements in the transformed texture feature data are the same. δ is a hyperparameter used to avoid the divisor being 0; k′ cij Represents the convolution kernel parameters at position (i, j) in the cth convolution kernel channel in the transformed convolution kernel.

[0100] The input feature map is convolved using the transformed convolution kernel to obtain an intermediate feature map for data fusion processing at this level.

[0101] Here, for example, the input feature map can be convolved using a transformed convolution kernel based on the following formula (3):

[0102]

[0103] Among them, f c,x+i,y+j Represents the feature value of the feature point at position (x+i,y+j) in the lth channel of the input feature map; represents the convolution kernel parameter at position (i, j) in the lth channel of the transformed convolution kernel; f′ lxy Represents the feature value of the feature point at position (x, y) in the lth channel in the intermediate feature map.

[0104] Through the above process, the first feature fusion of the transformed texture feature data and the input feature map is achieved.

[0105] In this way, using this method, since the convolution kernel is transformed by transforming the texture feature data, and the data amount of the convolution kernel is usually smaller than the data amount of the input feature map, it can greatly reduce the data that needs to be processed during the transformation process and improve data processing efficiency.

[0106] B:

[0107] For each level of data fusion processing in the multi-level data fusion processing, using the transformed texture feature data corresponding to the level of data fusion processing, the input feature map corresponding to the level of data fusion processing is transformed to obtain a transformed feature map;

[0108] The convolution kernel corresponding to the data fusion processing at this level is used to perform convolution processing on the transformed feature map to obtain an intermediate feature map corresponding to the data fusion processing at this level.

[0109] Here, the method of transforming the input feature map using the transformed texture feature data is similar to the method of transforming the convolution kernel using the transformed texture feature data in the above A, and will not be repeated here.

[0110] Following the above S202, the method for performing multi-level data fusion processing on the normal vector image and texture feature data provided by the embodiment of the present disclosure also includes:

[0111] S203: Perform a second feature fusion on the intermediate feature map of the data fusion processing at this level and the normal vector image to obtain a result feature map corresponding to the data fusion processing at this level.

[0112] In a specific implementation, when performing the second feature fusion on the intermediate feature map and the normal vector image of this level of data fusion processing, for example, the product between the intermediate feature map and the normal vector image can be calculated, and the product result matrix can be directly used as the result feature map corresponding to this level of data fusion processing.

[0113] Here, the product result matrix is ​​the result feature map, and the result feature map satisfies the following formula (4), for example:

[0114] f″ lxy =n xy f′ lxy (4)

[0115] f′ lxy represents the feature value of the feature point at position (x, y) in the lth channel of the intermediate feature map; f″ lxy Represents the feature value of the feature point at position (x, y) in the lth channel of the result feature map; n xy Represents the value of the normal vector corresponding to the pixel at position (x, y) in the normal vector image.

[0116] In addition, in another embodiment of the present disclosure, since the size of the normal vector image and the size of the intermediate feature map may be different, before the normal vector image and the intermediate feature map are subjected to the second feature fusion, the normal vector image is resized so that its size can meet the second feature fusion requirement with the intermediate feature map. Exemplarily, the normal vector image can be upsampled or downsampled to achieve the adjustment of its size. The specific adjustment method is related to the size of the intermediate feature map corresponding to each level of data fusion processing, and will not be repeated here.

[0117] In another embodiment, since the texture feature data of the face cannot fully express all the details of the face image, in the embodiment of the present disclosure, in order to make the finally generated reconstructed face image more realistic, after obtaining the product result matrix, based on the product result matrix corresponding to the data fusion processing at this level, and the preset deviation matrix and / or noise matrix of the data fusion processing at this level, a result feature map corresponding to the data fusion processing at this level can be obtained.

[0118] Exemplarily, the learning bias matrix here is a hyperparameter, and the noise matrix is, for example, a random Gaussian noise mapping.

[0119] Here, when obtaining the result feature graph corresponding to the data fusion processing at this level based on the product result matrix corresponding to the data fusion processing at this level, and the preset deviation matrix and / or noise matrix of the data fusion processing at this level, for example, the product result matrix corresponding to the data fusion processing at this level, and the preset deviation matrix and / or noise matrix of the data fusion processing at this level can be added in a bitwise manner to obtain the result feature graph corresponding to the data fusion processing at this level.

[0120] After obtaining the result feature map output by the last level of data fusion processing, the feature values ​​of each feature point in the result feature map can be directly used as the pixel values ​​of each pixel point in the reconstructed face image to render and generate the reconstructed face image.

[0121] In another embodiment of the present disclosure, the facial portrait generation method provided in the embodiment of the present disclosure is implemented using a pre-trained first target neural network.

[0122] Specifically, see Figure 3 As shown, the example of the present disclosure provides a specific method for training a first target neural network, including:

[0123] S301: Obtain a first sample normal vector image and first sample texture feature data of a first sample three-dimensional face model.

[0124] Here, the method for obtaining the first sample normal vector image and the first sample texture feature data of the first sample 3D face model is similar to the method for obtaining the normal vector image and texture feature data of the 3D face model in S101 above, and will not be repeated here.

[0125] S302: Using the first neural network to be trained, perform data fusion processing on the first sample normal vector image and the first sample texture feature data to obtain a reconstructed image generated by reconstructing the first sample three-dimensional face model.

[0126] Here, the process of generating the reconstructed image by the first neural network to be trained is similar to the process of generating the reconstructed face image mentioned above, and will not be repeated here.

[0127] S303: Based on the reconstructed image, a first training loss is obtained, and the first training loss is used to train the first neural network to be trained to obtain the first target neural network.

[0128] In a specific implementation, the first training loss includes at least one of the following: normal vector consistency loss, face key point consistency loss, and identity consistency loss.

[0129] (1): In the case where the first training loss includes a normal vector consistency loss, obtaining the first training loss based on the reconstructed image includes:

[0130] Perform normal vector prediction processing on the reconstructed image to obtain a predicted normal vector image of the reconstructed image.

[0131] Here, when performing normal vector prediction processing on the reconstructed image, for example, a three-dimensional face model may be first generated based on the reconstructed image, and then a predicted normal vector image of the reconstructed image may be obtained based on the generated three-dimensional face model.

[0132] The normal vector consistency loss is obtained by using the first sample normal vector image and the predicted normal vector image.

[0133] In a specific implementation, given a set of first sample normal vector images n, a set of first sample texture feature data z, and a set of random noise matrices ε, the reconstructed image I output by the first target neural network G out The following formula (5) is satisfied:

[0134] I out =G(n,z,ε) (5)

[0135] Use the face normal vector prediction network N to predict and generate I out Normal vector image n s , if the generated I out If the consistency with the first sample 3D face model is high enough, then I out Normal vector image z s The first sample normal vector image n of the first sample 3D face model should also be sufficiently consistent, so the normal vector consistency loss L n Satisfies the description formula (6):

[0136] L n =||P(I out )⊙(nN(I out ))||1 (6)

[0137] Where P(·) is the face detection network, which outputs the face region mask. The face region mask makes the normal vector consistency loss only effective in the face region; N(·) is the pre-trained face normal vector prediction network, which is used to predict I out Normal vector image n s ⊙ represents element-wise multiplication. Here, N(·) can adopt the SfSNet (Learning Shape, Reflectance and Illuminance of Faces in the Wild) network, for example.

[0138] (2): In the case where the first training loss includes key point consistency loss, obtaining the first training loss based on the reconstructed image includes:

[0139] performing key point recognition on the first reconstructed image to obtain a first key point of the first reconstructed image; and

[0140] Perform key point recognition based on the second reconstructed image to obtain second key points of the second reconstructed image;

[0141] The key point consistency loss is obtained by using the first key point and the second key point.

[0142] In this embodiment, the first sample texture feature data includes: first sample texture feature data and second sample texture feature data;

[0143] In a specific implementation, the reconstructed image includes a first reconstructed image obtained based on the first sample texture feature data, and a second reconstructed image obtained based on the second sample texture feature data; wherein the first sample normal vector images corresponding to the first reconstructed image and the second reconstructed image are the same.

[0144] Since the normal vector image focuses on the general structure of the face surface, and for the face, it is more concerned with whether the positions of the key points in the face correspond; in the embodiment of the present disclosure, given two different sets of first sample texture feature data, two different sets of first sample texture feature data and the same set of first sample normal vector images are used to generate a first reconstructed image and a second reconstructed image. In principle, the higher the matching degree between the generated face image and the first sample three-dimensional face model, the closer the positions of the face key points in the different reconstructed images generated using different texture feature data are. Therefore, in the embodiment of the present disclosure, the face key points are used as an important constraint condition, and the accuracy of the first target neural network is further improved by using the key point consistency loss between the first reconstructed image and the second reconstructed image.

[0145] Among them, the key point consistency loss L ldmkFor example, the following formula (7) is satisfied:

[0146] L ldmk (n,z1,z2)=||H(G(n,z1,ε))-H(G(n,z2,ε))||2 (7)

[0147] Wherein, z1 represents the first sample texture feature data; z2 represents the target face texture feature data; H(·) represents key point recognition of the image.

[0148] (3): For the case where the first training loss includes identity consistency loss,

[0149] The first sample normal vector image of the first sample 3D face model includes: a first sample normal vector image and a second sample normal vector image;

[0150] The facial expressions and / or facial postures corresponding to the first sample normal vector image and the second sample normal vector image are different;

[0151] The method of using the first neural network to be trained to perform data fusion processing on the first sample normal vector image and the first sample texture feature data to obtain a reconstructed image generated by reconstructing the first sample three-dimensional face model includes:

[0152] Using the first neural network to be trained, performing data fusion processing on the first sample normal vector image and the first sample texture feature data to obtain a third reconstructed image of the first sample three-dimensional face model; and

[0153] Using the first neural network to be trained, data fusion processing is performed on the second sample normal vector image and the first sample texture feature data to obtain a fourth reconstructed image of the first sample three-dimensional face model;

[0154] The obtaining a first training loss based on the reconstructed image includes:

[0155] Performing identity recognition on the third reconstructed image to obtain a first identity recognition result; and performing identity recognition on the fourth reconstructed image to obtain a second identity recognition result;

[0156] The identity consistency loss is obtained based on the first identity recognition result and the second identity recognition result.

[0157] In a specific implementation, when the facial posture or facial expression of the same person changes, but the face does not change, it should be possible to identify that images with different facial postures or facial expressions are images of the same person. Therefore, in the disclosed embodiment, the first sample normal vector image of the first sample three-dimensional face model includes: a first sample normal vector image, and a second sample normal vector image; wherein the facial expressions and / or facial postures corresponding to the first sample normal vector image and the second sample normal vector image are different. The reconstructed image obtained includes a third reconstructed image obtained based on the first sample normal vector image and a fourth reconstructed image obtained based on the second sample normal vector image, and then identity recognition results are performed on the third reconstructed image and the fourth reconstructed image, respectively, and then the identity recognition results of the two are used to obtain identity consistency loss.

[0158] For example, the identity consistency loss L id For example, the following formula (8) is satisfied:

[0159]

[0160] Among them, α represents the face shape; β1 and β2 represent different facial expressions; θ1 and θ2 represent different facial postures; R(·) represents the identity recognition processing of the image.

[0161] In this way, by keeping z unchanged and using different expressions and postures to generate consistent facial identities, that is, the features obtained through face recognition are similar, the decoupling of texture feature data z and expression shape parameters is completed.

[0162] (4): For the case where the first training loss includes adversarial loss,

[0163] When training the first target neural network, an adversarial network of the first target neural network can be constructed; in the process of training the first target neural network, the reconstructed image predicted by the first neural network to be trained is used as the input of the adversarial network, and the adversarial network is used to predict the authenticity of the reconstructed image; the loss generated by the adversarial network, i.e., the adversarial loss L adv .

[0164] This introduces the adversarial loss L adv As a part of the first loss for training the first target neural network, the authenticity of the reconstructed face image generated by the first target neural network is improved.

[0165] In one embodiment of the present disclosure, the first loss provided for the embodiment of the present disclosure includes the above-mentioned normal vector consistency loss, face key point consistency loss, identity consistency loss, and multiple losses in the adversarial loss. The multiple losses can be weighted summed as the first loss.

[0166] Exemplarily, the loss L when training the first target neural network is GAR The following formula (9) is satisfied:

[0167] L GAR =λ n L n +λ ldmk L ldmk +λ id L id +λ adv L adv (9)

[0168] Among them, λ represents the weight of the corresponding loss.

[0169] See also Figure 4 As shown, the embodiment of the present disclosure also provides an example of a specific structure of a first target neural network GAR, and the first target neural network GAR includes: N rendering blocks.

[0170] The first fully connected network is used to perform a first fully connected processing on the texture feature data z to obtain first intermediate texture feature data.

[0171] The second fully connected network is used to perform a second fully connected processing on the first intermediate texture feature data to obtain the transformed texture feature data w corresponding to the corresponding rendering block.

[0172] Noise propagation module: used to propagate the noise matrix ε to each rendering block.

[0173] The up-sampling module is used to resize the normal vector image n so as to adjust the normal vector to a size required by each rendering block.

[0174] Above Figure 4 A reconstructed face image with high authenticity can be obtained based on the texture feature data z, the normal vector image n and the noise matrix ε.

[0175] See also Figure 5 As shown, the example of the present disclosure also provides a specific structure of a rendering block, which includes:

[0176] The convolution kernel transformation module is used to transform the convolution kernel k using the transformed texture feature data w to obtain the transformed convolution kernel k′.

[0177] Upsampling layer: used to upsample the result feature map output by the previous rendering block, or the preset feature map, to obtain the input feature map corresponding to the current rendering block.

[0178] Convolution layer: used to perform convolution processing on the input feature map corresponding to the current rendering block using the transformed convolution kernel k′ to obtain the intermediate feature map corresponding to the current rendering block.

[0179] Fusion module: used to perform second feature fusion on the intermediate feature map and the normal vector image to obtain a fused feature map.

[0180] Then the fused feature map, the noise matrix ε, and the deviation matrix b are added in place to obtain the result feature map corresponding to the current rendering block.

[0181] The following is an explanation of the three-dimensional model generation method provided by the embodiment of the present disclosure.

[0182] See also Figure 6 As shown, the embodiment of the present disclosure provides a method for generating a three-dimensional face model, comprising:

[0183] S601: Performing three-dimensional face reconstruction and texture feature recovery on a target face image including a target face, to obtain an initial normal vector image of the target face and initial texture feature data of the target face.

[0184] S602: Obtaining a three-dimensional face model of the target face based on the initial normal vector image and the initial texture feature data.

[0185] The disclosed embodiment predicts an initial normal vector image of the target face based on a target facial image of the target face, and then obtains a three-dimensional facial model of the target face based on the initial normal vector image and the initial texture feature data. In this method, since the facial image based on the initial normal vector image and the initial texture feature data has a higher accuracy, the generated three-dimensional facial model of the target face also has a higher precision.

[0186] The above S601 to S602 are described in detail below.

[0187] Regarding the above S601:

[0188] When performing three-dimensional face reconstruction on a target face image of a target face and obtaining an initial normal vector image of the target face, a pre-trained three-dimensional model prediction network is used to process the target face image to obtain a three-dimensional face model of the target face image, and then the three-dimensional face model of the target face image is used to determine the initial normal vector image of the target face.

[0189] Exemplarily, for example, a 3DMM algorithm may be used to perform three-dimensional face reconstruction on a target face image.

[0190] When performing texture feature restoration on a target facial image including a target face to obtain initial texture feature data of the target face, for example, a pre-trained second target neural network may be used to perform texture feature restoration on the target facial image to obtain initial texture feature data of the target face.

[0191] In the embodiment of the present disclosure, the second target neural network can be obtained by training the first target neural network provided by the embodiment of the present disclosure, for example.

[0192] In a specific implementation, when training the second target neural network, for example, it can be obtained by coupling training with the adversarial first target neural network. For example, the network structure of the second target neural network can be designed as the inverse structure of the first target neural network, that is, the second target neural network includes multiple inverse rendering blocks; in each inverse rendering block, a convolutional layer is used to replace the upsampling layer in each rendering block in the first target neural network, and an upsampling layer is used to replace the convolutional layer in the rendering block, and each inverse rendering block can output an output feature map; the output feature maps of each level of inverse rendering blocks are used to obtain the predicted initial texture feature data.

[0193] In the second target neural network, the output feature map of each network layer has the same size as the input feature map of the network layer corresponding to the first target neural network.

[0194] When using the second target neural network to restore the initial texture feature data of the target face image, for example, the output feature map of each inverse rendering block in the second target neural network can be obtained; then for each inverse rendering block, the mean and variance of the feature value of the feature point in the output feature map of the inverse rendering block are calculated; then the means and variances corresponding to each inverse rendering block are combined to obtain a target feature map, and the target feature map is fully connected using a fully connected network to obtain initial texture feature data.

[0195] The present disclosure also provides a specific method for training a second target neural network, including:

[0196] Using the first target neural network, processing the second sample normal vector image and the second sample texture feature data of the sample three-dimensional face model to obtain a sample face image of the sample three-dimensional face model;

[0197] Processing the sample face image using a second neural network to be trained to obtain predicted texture feature data corresponding to the sample face image;

[0198] Determining a third loss based on the predicted texture feature data and the second sample texture feature data;

[0199] Based on the third loss, the second neural network to be trained is trained to obtain a second target neural network.

[0200] Here, the method for obtaining the predicted texture feature data is similar to the method for obtaining the initial texture feature data, and will not be described in detail here.

[0201] Exemplarily, when training the second target neural network, the loss function of the third loss satisfies the following formula (10):

[0202]

[0203] Where R represents the second target neural network; I out Represents the output of the first target neural network; it satisfies: I out =G(n,z,ε); When training the second target neural network, I out Sample face images used as input to the second target neural network to be trained. i (I out ) represents the output feature map of the i-th inverse rendering block of the second target neural network; G i represents the input texture feature data w of the i-th rendering block of the first target neural network; μ and σ represent the mean and standard deviation, respectively.

[0204] MLP([μ(R i (I out ));σ(R i (I out ))]) represents the predicted texture feature data obtained by performing full connection processing on the target feature map using the fully connected layer MLP; z represents the second sample texture feature data; G i (n,z,θ) represents the result feature map output by the i-th rendering block of the first target neural network; R i (I out ) represents the output feature map of the i-th inverse rendering block output of the second target neural network (to be trained).

[0205] After the second target neural network is trained, the target face image can be processed using the second target neural network to obtain the initial normal vector image and initial texture feature data corresponding to the target face image.

[0206] With respect to the above S602: when obtaining the three-dimensional face model of the target face based on the initial normal vector image and the initial texture feature data, for example, the following method may be used:

[0207] Using the initial normal vector image as the current normal vector image and the initial texture feature data as the current texture feature data,

[0208] A current reconstructed face image is generated based on the current normal vector image and the current texture feature data.

[0209] Here, when generating the current reconstructed facial image based on the current normal vector image and the current texture feature data, for example, the current normal vector image can be used as the normal vector image, and the current texture feature data can be used as the texture feature data, and the facial image generation method provided by any embodiment of the present disclosure can be used to generate a reconstructed facial image; and the reconstructed facial image can be used as the current reconstructed facial image generated based on the current normal vector image and the current reconstructed facial image.

[0210] For details, please refer to the above Figure 1 to Figure 5 The corresponding embodiments are shown and will not be described again here.

[0211] Based on the target face image and the current reconstructed face image, the current normal vector is adjusted to obtain a target normal vector image of the target face.

[0212] Here, the following iterative process is performed until the second loss is less than a preset loss threshold:

[0213] Obtaining a second loss based on the target face image and the current reconstructed face image;

[0214] Using the second loss to adjust the current normal vector image and the current texture feature data to obtain a new normal vector image and new texture feature data;

[0215] The new normal vector image is used as the current normal vector image, and the new texture feature data is used as the current texture feature data, and the process returns to the step of generating the current reconstructed face image based on the current normal vector image and the current texture feature data.

[0216] After the second loss is less than a preset loss threshold, the current normal vector image in the last iteration process is used as the target normal vector image.

[0217] After the target normal vector image is obtained, the target normal vector image can be used to generate a three-dimensional face model of the target face.

[0218] In the above embodiment, the second loss includes, for example: pixel consistency loss and / or classification feature consistency loss.

[0219] In which, when the second loss includes pixel consistency loss, for example, the difference between the pixel value of the current reconstructed face image and the pixel value of the target face image is calculated, and the L2 norm of the difference is calculated, and the pixel consistency loss is determined based on the result of the L2 norm.

[0220] In the case where the second loss includes classification feature consistency loss, for example, the target face image can be classified using a pre-trained image classification network to obtain first feature data output by a target network layer of the image classification network; and

[0221] Using the image classification network to classify the current reconstructed face image, and obtain the second feature data output by the target network layer;

[0222] Based on the first feature data and the second feature data, the classification feature consistency loss is obtained; wherein the image classification network is trained using the target face image and the current reconstructed face image.

[0223] The specific location of the target network layer can be determined according to the actual situation. For example, the target network layer is, for example, the 1st to 5th network layers in the image classification network.

[0224] Exemplarily, the loss function of the second loss satisfies the following formula (11):

[0225]

[0226] Where, G represents the first target neural network; is the normal vector image generated by using α, β, θ; where α, β, θ have the same meaning as the above formula (8); I t represents the target face image; represents pixel consistency loss; F represents the classification network trained using the target face image and the current reconstructed face image; F i Represents the feature data output by the i-th network layer of the classification network; represents the classification feature consistency loss; λ n represents the weight of the regularization term on random noise; minimize the above second loss L f , and obtain the optimized geometric parameters α, β and θ. Based on the optimized geometric parameters α, β and θ, the target normal vector image of the target face can be determined.

[0227] In this way, by keeping the target face image and the current reconstructed face image consistent in classification features as supervision of the optimization process, the obtained three-dimensional face model has a higher match with the target face, so the three-dimensional face model has higher accuracy.

[0228] Those skilled in the art will appreciate that, in the above method of specific implementation, the order in which the steps are written does not imply a strict execution order and does not constitute any limitation on the implementation process. The specific execution order of the steps should be determined by their functions and possible internal logic.

[0229] Based on the same inventive concept, the embodiment of the present disclosure also provides a facial image generating device corresponding to the facial image generating method. Since the principle of solving the problem by the device in the embodiment of the present disclosure is similar to the above-mentioned facial image generating method in the embodiment of the present disclosure, the implementation of the device can refer to the implementation of the method, and the repeated parts will not be repeated.

[0230] Reference Figure 7 FIG. 1 is a schematic diagram of a facial image generating device provided by an embodiment of the present disclosure, wherein the device comprises:

[0231] A first acquisition module 71 is used to acquire a normal vector image and texture feature data; wherein the pixel value of each pixel point in the normal vector image represents the value of the normal vector of the model vertex corresponding to the pixel point in the three-dimensional face model corresponding to the normal vector image;

[0232] The first processing module 72 is used to perform multi-level data fusion processing on the normal vector image and the texture feature data to obtain a reconstructed face image.

[0233] In one possible implementation, when the first processing module 72 performs multi-level data fusion processing on the normal vector image and the texture feature data to obtain a reconstructed facial image, the module is used to: perform feature transformation on the texture feature data to obtain transformed texture feature data corresponding to the multi-level data fusion processing; for each level of data fusion processing in the multi-level data fusion processing, perform a first feature fusion on the transformed texture feature data corresponding to the data fusion processing at that level and the input feature map corresponding to the data fusion processing at that level to obtain an intermediate feature map of the data fusion processing at that level; perform a second feature fusion on the intermediate feature map of the data fusion processing at that level and the normal vector image to obtain a result feature map corresponding to the data fusion processing at that level; and obtain the reconstructed facial image based on the result feature map corresponding to the last level of data fusion processing.

[0234] In one possible implementation, when the first processing module 72 performs feature transformation on the texture feature data to obtain transformed texture feature data corresponding to the multi-level data fusion processing, it is used to: perform a first full connection processing on the texture feature data to obtain first intermediate texture feature data; perform multiple second full connection processing on the first intermediate texture feature data to obtain transformed texture feature data corresponding to the multi-level data fusion processing; wherein the full connection parameters of different second full connection processing are different.

[0235] In one possible implementation, the first processing module 72, before performing a first feature fusion on the transformed texture feature data corresponding to each level of data fusion processing in the multi-level data fusion processing and the input feature map corresponding to the data fusion processing at that level to obtain the intermediate feature map of the data fusion processing at that level, is also used for: in the case where the data fusion processing at that level is the first level of data fusion processing, upsampling the preset feature map to obtain the input feature map corresponding to the data fusion processing at that level; in the case where the data fusion processing at that level is not the first level of data fusion processing, upsampling the result feature map output by the previous level of fusion processing of the data fusion processing at that level to obtain the input feature map corresponding to the data fusion processing at that level.

[0236] In one possible implementation, the first processing module 72, for each level of data fusion processing in the multi-level data fusion processing, performs a first feature fusion on the transformed texture feature data corresponding to the level of data fusion processing and the input feature map corresponding to the level of data fusion processing to obtain an intermediate feature map of the data fusion processing at that level. The first processing module 72 is used to: for each level of data fusion processing in the multi-level data fusion processing, transform the convolution kernel corresponding to the level of fusion processing using the transformed texture feature data corresponding to the level of data fusion processing to obtain a transformed convolution kernel; and perform convolution processing on the input feature map using the transformed convolution kernel to obtain an intermediate feature map of the data fusion processing at that level.

[0237] In one possible implementation, the first processing module 72, for each level of data fusion processing in the multi-level data fusion processing, performs a first feature fusion on the transformed texture feature data corresponding to the level of data fusion processing and the input feature map corresponding to the level of data fusion processing to obtain an intermediate feature map of the data fusion processing at that level. The first processing module 72 is used to: for each level of data fusion processing in the multi-level data fusion processing, transform the input feature map corresponding to the level of fusion processing using the transformed texture feature data corresponding to the level of data fusion processing to obtain a transformed feature map; and perform convolution processing on the transformed feature map using the convolution kernel corresponding to the level of data fusion processing to obtain an intermediate feature map corresponding to the level of data fusion processing.

[0238] In one possible implementation, the first processing module 72, when performing a second feature fusion on the intermediate feature map of the data fusion processing at this level and the normal vector image to obtain a result feature map corresponding to the data fusion processing at this level, is used to: determine a product result matrix of the intermediate feature map of the data fusion processing at this level and the normal vector image; and determine the result feature map corresponding to the data fusion processing at this level based on the product result matrix.

[0239] In one possible implementation, the first processing module 72, when determining the result feature graph corresponding to the data fusion processing at this level based on the product result matrix, is used to: obtain the result feature graph corresponding to the data fusion processing at this level based on the product result matrix corresponding to the data fusion processing at this level, and the preset deviation matrix and / or noise matrix of the data fusion processing at this level.

[0240] In one possible implementation, the first processing module 72, when obtaining a result feature graph corresponding to the data fusion processing at this level based on the product result matrix corresponding to the data fusion processing at this level, and the preset deviation matrix and / or noise matrix for the data fusion processing at this level, is used to: perform positional addition on the product result matrix corresponding to the data fusion processing at this level, and the preset deviation matrix and / or noise matrix for the data fusion processing at this level, to obtain a result feature graph corresponding to the data fusion processing at this level.

[0241] In a possible implementation manner, the facial image generation method is implemented by a pre-trained first target neural network.

[0242] In a possible implementation, it also includes: a first training module 73, which is used to obtain the first target neural network by the following method: obtaining a first sample normal vector image and first sample texture feature data of a first sample three-dimensional face model; using the first neural network to be trained, performing data fusion processing on the first sample normal vector image and the first sample texture feature data to obtain a reconstructed image of the first sample three-dimensional face model; based on the reconstructed image, obtaining a first training loss, and using the first training loss to train the first neural network to be trained to obtain the first target neural network.

[0243] In a possible implementation, the first training loss includes at least one of the following: normal vector consistency loss, facial key point consistency loss, identity consistency loss, and adversarial loss.

[0244] In one possible implementation, the first training loss includes a normal vector consistency loss. The first training module 73, when obtaining the first training loss based on the reconstructed image, is used to: perform normal vector prediction processing on the reconstructed image to obtain a predicted normal vector image of the reconstructed image; and obtain the normal vector consistency loss using the first sample normal vector image and the predicted normal vector image.

[0245] In a possible implementation, the first training loss includes a key point consistency loss, and the first sample texture feature data includes: first sample texture feature data and second sample texture feature data; the reconstructed image includes a first reconstructed image obtained based on the first sample texture feature data, and a second reconstructed image obtained based on the second sample texture feature data; the first training module 73, when obtaining the first training loss based on the reconstructed image, is used to: perform key point recognition on the first reconstructed image to obtain a first key point of the first reconstructed image; and perform key point recognition based on the second reconstructed image to obtain a second key point of the second reconstructed image; and obtain the key point consistency loss using the first key point and the second key point.

[0246] In one possible implementation manner, the first training loss includes identity consistency loss, and the first sample normal vector image of the first sample 3D face model includes: a first sample normal vector image, and a second sample normal vector image; wherein the first sample normal vector image and the second sample normal vector image correspond to different facial expressions and / or facial postures; the first training module 73, when using the first neural network to be trained to perform data fusion processing on the first sample normal vector image and the first sample texture feature data to obtain a reconstructed image generated by reconstructing the first sample 3D face model, is used to: use the first neural network to be trained to perform data fusion processing on the first sample normal vector image and the first sample texture feature data The first sample texture feature data is subjected to data fusion processing to obtain a third reconstructed image of the first sample three-dimensional face model; and the second sample normal vector image and the first sample texture feature data are subjected to data fusion processing by using the first neural network to be trained to obtain a fourth reconstructed image of the first sample three-dimensional face model; the first training module 73, when obtaining the first training loss based on the reconstructed image, is used to: perform identity recognition on the third reconstructed image to obtain a first identity recognition result; and perform identity recognition on the fourth reconstructed image to obtain a second identity recognition result; and obtain the identity consistency loss based on the first identity recognition result and the second identity recognition result.

[0247] For descriptions of the processing flow of each module in the device and the interaction flow between each module, reference may be made to the relevant descriptions in the above method embodiment, which will not be described in detail here.

[0248] See also Figure 8 As shown, the embodiment of the present disclosure also provides a 3D face model generation device, including:

[0249] A second acquisition module 81 is used to perform three-dimensional face reconstruction and texture feature recovery on a target face image including a target face, and obtain an initial normal vector image of the target face and initial texture feature data of the target face;

[0250] The second processing module 82 is used to obtain a three-dimensional face model of the target face based on the initial normal vector image and the initial texture feature data.

[0251] In one possible implementation, when the three-dimensional face model of the target face is obtained based on the initial normal vector image and the initial texture feature data, the second processing module 82 is used to: use the initial normal vector image as the current normal vector image and the initial texture feature data as the current texture feature data, and generate a current reconstructed face image based on the current normal vector image and the current texture feature data; adjust the current normal vector based on the target face image and the current reconstructed face image to obtain a target normal vector image of the target face; and generate a three-dimensional face model of the target face based on the target normal vector image.

[0252] In one possible implementation, when generating a reconstructed facial image based on the current normal vector image and the texture feature data, the second processing module 82 is used to: use the current normal vector image as the normal vector image and the current texture feature data as the texture feature data to generate a reconstructed facial image using the facial image generation method described in any one of the first aspects; and use the reconstructed facial image as the current reconstructed facial image generated based on the current normal vector image and the current reconstructed facial image.

[0253] In one possible implementation, the second processing module 82, when adjusting the current normal vector based on the target face image and the currently reconstructed face image to obtain the target normal vector image of the target face, is used to: execute the following iterative process until the second loss is less than a preset loss threshold: obtain a second loss based on the target face image and the currently reconstructed face image; use the second loss to adjust the current normal vector image and the current texture feature data to obtain a new normal vector image and new texture feature data; use the new normal vector image as the current normal vector image and the new texture feature data as the current texture feature data, and return to the step of generating the current reconstructed face image based on the current normal vector image and the current texture feature data.

[0254] In a possible implementation, the second loss includes: pixel consistency loss and / or classification feature consistency loss.

[0255] In one possible implementation, for the case where the second loss includes classification feature consistency loss, the second processing module 82, when obtaining the second loss based on the target face image and the current reconstructed face image, is used to: use a pre-trained image classification network to classify the target face image to obtain first feature data output by a target network layer of the image classification network; and use the image classification network to classify the current reconstructed face image to obtain second feature data output by the target network layer; obtain the classification feature consistency loss based on the first feature data and the second feature data; wherein the image classification network is trained using the target face image and the current reconstructed face image.

[0256] In one possible implementation, when the second acquisition module 81 performs texture feature restoration on a target facial image including a target face to obtain initial texture feature data of the target face, it is used to: perform texture feature restoration on the target facial image using a pre-trained second target neural network to obtain an initial normal vector image of the target face; wherein the second target neural network is trained using the first target neural network described in any embodiment of the present disclosure.

[0257] In a possible implementation, it also includes: a second training module 83, which is used to train the second target neural network in the following manner: using the first target neural network, processing the second sample normal vector image and the second sample texture feature data of the sample three-dimensional face model to obtain a sample face image of the sample three-dimensional face model; using the second neural network to be trained to process the sample face image to obtain predicted texture feature data corresponding to the sample face image; determining a third loss based on the predicted texture feature data and the second sample texture feature data; based on the third loss, training the second neural network to be trained to obtain a second target neural network.

[0258] The present disclosure also provides an electronic device, such as Fig. 9 FIG. 1 is a schematic diagram of the structure of an electronic device provided in an embodiment of the present disclosure, including:

[0259] A processor 91 and a memory 92; the memory 92 stores machine-readable instructions executable by the processor 91, and the processor 91 is used to execute the machine-readable instructions stored in the memory 92. When the machine-readable instructions are executed by the processor 91, the processor 91 performs the following steps:

[0260] Acquire a normal vector image and texture feature data; wherein the pixel value of each pixel point in the normal vector image represents the value of the normal vector of the model vertex corresponding to the pixel point in the three-dimensional face model corresponding to the normal vector image;

[0261] Multi-level data fusion processing is performed on the normal vector image and the texture feature data to obtain a reconstructed face image.

[0262] Or perform the following steps: perform three-dimensional face reconstruction and texture feature recovery on a target face image including a target face, and obtain an initial normal vector image of the target face and initial texture feature data of the target face;

[0263] Based on the initial normal vector image and the initial texture feature data, a three-dimensional face model of the target face is obtained.

[0264] The above-mentioned memory 92 includes internal memory 921 and external memory 922; the memory 921 here is also called internal memory, which is used to temporarily store the calculation data in the processor 91 and the data exchanged with the external memory 922 such as the hard disk. The processor 91 exchanges data with the external memory 922 through the internal memory 921.

[0265] The specific execution process of the above instructions can refer to the steps of the facial image generation method or the three-dimensional facial model generation method described in the embodiments of the present disclosure, which will not be repeated here.

[0266] The present disclosure also provides a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, the steps of the face image generation method or the three-dimensional face model generation method described in the above method embodiment are executed. The storage medium can be a volatile or non-volatile computer-readable storage medium.

[0267] The embodiments of the present disclosure also provide a computer program product, which carries a program code. The instructions included in the program code can be used to execute the steps of the facial image generation method or the three-dimensional facial model generation method described in the above method embodiments. For details, please refer to the above method embodiments, which will not be repeated here.

[0268] The computer program product may be implemented in hardware, software or a combination thereof. In one optional embodiment, the computer program product is implemented as a computer storage medium. In another optional embodiment, the computer program product is implemented as a software product, such as a software development kit (SDK).

[0269] Those skilled in the art can clearly understand that, for the convenience and simplicity of description, the specific working process of the system and device described above can refer to the corresponding process in the aforementioned method embodiment, and will not be repeated here. In the several embodiments provided in the present disclosure, it should be understood that the disclosed system, device and method can be implemented in other ways. The device embodiments described above are merely schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some communication interfaces, and the indirect coupling or communication connection of the device or unit can be electrical, mechanical or other forms.

[0270] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0271] In addition, each functional unit in each embodiment of the present disclosure may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0272] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a non-volatile computer-readable storage medium that is executable by a processor. Based on this understanding, the technical solution of the present disclosure, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions for enabling an electronic device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present disclosure. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0273] Finally, it should be noted that the above-described embodiments are only specific implementation methods of the present disclosure, which are used to illustrate the technical solutions of the present disclosure, rather than to limit them. The protection scope of the present disclosure is not limited thereto. Although the present disclosure is described in detail with reference to the above-described embodiments, ordinary technicians in the field should understand that any technician familiar with the technical field can still modify the technical solutions recorded in the above-described embodiments within the technical scope disclosed in the present disclosure, or can easily think of changes, or make equivalent replacements for some of the technical features therein; and these modifications, changes or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present disclosure, and should be included in the protection scope of the present disclosure. Therefore, the protection scope of the present disclosure should be based on the protection scope of the claims.

Claims

1. A method for generating a face image, characterized in that: include: Acquire a normal vector image and texture feature data; wherein the pixel value of each pixel point in the normal vector image represents the value of the normal vector of the model vertex corresponding to the pixel point in the three-dimensional face model corresponding to the normal vector image; Performing multi-level data fusion processing on the normal vector image and the texture feature data to obtain a reconstructed face image; The performing multi-level data fusion processing on the normal vector image and the texture feature data to obtain a reconstructed face image includes: Performing feature transformation on the texture feature data to obtain transformed texture feature data corresponding to the multi-level data fusion processing; For each level of data fusion processing in the multi-level data fusion processing, performing a first feature fusion on the transformed texture feature data corresponding to the level of data fusion processing and the input feature map corresponding to the level of data fusion processing to obtain an intermediate feature map of the level of data fusion processing; Performing a second feature fusion on the intermediate feature map of the data fusion processing at this level and the normal vector image to obtain a result feature map corresponding to the data fusion processing at this level; Based on the corresponding result feature map of the last level of data fusion processing, the reconstructed face image is obtained.

2. The method for generating a facial image according to claim 1, characterized in that: The step of performing feature transformation on the texture feature data to obtain transformed texture feature data corresponding to the multi-level data fusion processing includes: Performing a first full connection process on the texture feature data to obtain first intermediate texture feature data; The first intermediate texture feature data is subjected to multiple second full connection processing to obtain transformed texture feature data corresponding to the multi-level data fusion processing respectively; wherein the full connection parameters of different second full connection processing are different.

3. The method for generating a facial image according to claim 1 or 2, characterized in that: For each level of data fusion processing in the multi-level data fusion processing, before performing a first feature fusion on the transformed texture feature data corresponding to the level of data fusion processing and the input feature map corresponding to the level of data fusion processing to obtain an intermediate feature map of the level of data fusion processing, the method further includes: In the case where the data fusion processing at this level is the first level of data fusion processing, upsampling the preset feature map is performed to obtain an input feature map corresponding to the data fusion processing at this level; In case that the data fusion processing at this level is not the first level data fusion processing, the result feature map output by the data fusion processing at the previous level of the data fusion processing is upsampled to obtain the input feature map corresponding to the data fusion processing at this level.

4. The method for generating a facial image according to any one of claims 1 to 3, characterized in that: For each level of data fusion processing in the multi-level data fusion processing, performing a first feature fusion on the transformed texture feature data corresponding to the level of data fusion processing and the input feature map corresponding to the level of data fusion processing to obtain an intermediate feature map of the level of data fusion processing, including: For each level of data fusion processing in the multi-level data fusion processing, using the transformed texture feature data corresponding to the level of data fusion processing, transform the convolution kernel corresponding to the level of fusion processing to obtain the transformed convolution kernel; The input feature map is convolved using the transformed convolution kernel to obtain an intermediate feature map for data fusion processing at this level.

5. The method for generating a facial image according to any one of claims 1 to 4, characterized in that: For each level of data fusion processing in the multi-level data fusion processing, performing a first feature fusion on the transformed texture feature data corresponding to the level of data fusion processing and the input feature map corresponding to the level of data fusion processing to obtain an intermediate feature map of the level of data fusion processing, including: For each level of data fusion processing in the multi-level data fusion processing, using the transformed texture feature data corresponding to the level of data fusion processing, transform the input feature map corresponding to the level of fusion processing to obtain a transformed feature map; The transformed feature map is convolved using the convolution kernel corresponding to the data fusion processing at this level to obtain an intermediate feature map corresponding to the data fusion processing at this level.

6. The method for generating a facial image according to any one of claims 1 to 5, characterized in that: The performing second feature fusion on the intermediate feature map of the data fusion processing at this level and the normal vector image to obtain a result feature map corresponding to the data fusion processing at this level includes: Determine a product result matrix of an intermediate feature map of the data fusion processing at this level and the normal vector image; A result feature map corresponding to the data fusion processing at this level is determined based on the product result matrix.

7. The method for generating a facial image according to claim 6, characterized in that: The determining of the result feature graph corresponding to the data fusion processing at this level based on the product result matrix includes: Based on the product result matrix corresponding to the data fusion processing at this level, and the preset deviation matrix and / or noise matrix of the data fusion processing at this level, a result feature map corresponding to the data fusion processing at this level is obtained.

8. The method for generating a facial image according to claim 7, characterized in that: The method of obtaining a result feature map corresponding to the data fusion processing at this level based on the product result matrix corresponding to the data fusion processing at this level, and the preset deviation matrix and / or noise matrix of the data fusion processing at this level, comprises: The product result matrix corresponding to the data fusion processing at this level, and the preset deviation matrix and / or noise matrix of the data fusion processing at this level are added in parallel to obtain the result feature map corresponding to the data fusion processing at this level.

9. The method for generating a facial image according to any one of claims 1 to 8, characterized in that: The facial image generation method is implemented by a pre-trained first target neural network.

10. The method for generating a facial image according to claim 9, characterized in that: The first target neural network is obtained in the following manner: Acquire a first sample normal vector image and first sample texture feature data of a first sample three-dimensional face model; Using the first neural network to be trained, performing data fusion processing on the first sample normal vector image and the first sample texture feature data to obtain a reconstructed image of the first sample three-dimensional face model; Based on the reconstructed image, a first training loss is obtained, and the first neural network to be trained is trained using the first training loss to obtain the first target neural network.

11. The method for generating a facial image according to claim 10, characterized in that: The first training loss includes at least one of the following: normal vector consistency loss, face key point consistency loss, identity consistency loss and adversarial loss.

12. The method for generating a facial image according to claim 11, characterized in that: The first training loss includes a normal vector consistency loss, and obtaining the first training loss based on the reconstructed image includes: Performing normal vector prediction processing on the reconstructed image to obtain a predicted normal vector image of the reconstructed image; The normal vector consistency loss is obtained by using the first sample normal vector image and the predicted normal vector image.

13. The method for generating a facial image according to claim 11 or 12, characterized in that: The first training loss includes key point consistency loss, the first sample texture feature data includes: first sample texture feature data and second sample texture feature data; the reconstructed image includes a first reconstructed image obtained based on the first sample texture feature data, and a second reconstructed image obtained based on the second sample texture feature data; The obtaining a first training loss based on the reconstructed image includes: Performing key point recognition on the first reconstructed image to obtain a first key point of the first reconstructed image; Perform key point recognition based on the second reconstructed image to obtain second key points of the second reconstructed image; The key point consistency loss is obtained by using the first key point and the second key point.

14. The method for generating a facial image according to any one of claims 11 to 13, characterized in that: The first training loss includes identity consistency loss, and the first sample normal vector image of the first sample 3D face model includes: a first sample normal vector image and a second sample normal vector image; The first sample normal vector image and the second sample normal vector image correspond to different facial expressions and / or facial postures; The method of using the first neural network to be trained to perform data fusion processing on the first sample normal vector image and the first sample texture feature data to obtain a reconstructed image generated by reconstructing the first sample three-dimensional face model includes: Using the first neural network to be trained, performing data fusion processing on the first sample normal vector image and the first sample texture feature data to obtain a third reconstructed image of the first sample three-dimensional face model; and Using the first neural network to be trained, performing data fusion processing on the second sample normal vector image and the first sample texture feature data to obtain a fourth reconstructed image of the first sample three-dimensional face model; The obtaining a first training loss based on the reconstructed image includes: Performing identity recognition on the third reconstructed image to obtain a first identity recognition result; and performing identity recognition on the fourth reconstructed image to obtain a second identity recognition result; The identity consistency loss is obtained based on the first identity recognition result and the second identity recognition result.

15. A method for generating a three-dimensional face model, characterized in that: include: Performing three-dimensional face reconstruction and texture feature recovery on the target face image to obtain an initial normal vector image of the target face and initial texture feature data of the target face; Based on the initial normal vector image and the initial texture feature data, a three-dimensional face model of the target face is obtained; The step of obtaining a three-dimensional face model of the target face based on the initial normal vector image and the initial texture feature data comprises: Using the initial normal vector image as the current normal vector image and the initial texture feature data as the current texture feature data, Generate a current reconstructed face image based on the current normal vector image and the current texture feature data; Based on the target face image and the current reconstructed face image, adjusting the current normal vector image to obtain a target normal vector image of the target face; Based on the target normal vector image, generating a three-dimensional face model of the target face; The generating a reconstructed face image based on the current normal vector image and the texture feature data comprises: Using the current normal vector image as the normal vector image and the current texture feature data as the texture feature data, and using the face image generation method according to any one of claims 1 to 14 to generate a reconstructed face image; The reconstructed facial image is used as the current reconstructed facial image generated based on the current normal vector image and the current reconstructed facial image.

16. The method for generating a three-dimensional face model according to claim 15, characterized in that: The adjusting the current normal vector image based on the target face image and the current reconstructed face image to obtain a target normal vector image of the target face includes: The following iterative process is performed until the second loss is less than the preset loss threshold: Obtaining a second loss based on the target face image and the current reconstructed face image; Using the second loss to adjust the current normal vector image and the current texture feature data to obtain a new normal vector image and new texture feature data; Using the new normal vector image as the current normal vector image and the new texture feature data as the current texture feature data, and returning to the step of generating a current reconstructed face image based on the current normal vector image and the current texture feature data; The current normal vector image in the last iteration process is used as the target normal vector image.

17. The method for generating a face model according to claim 16, characterized in that: The second loss includes: pixel consistency loss and / or classification feature consistency loss.

18. The method for generating a face model according to claim 17, characterized in that: The second loss includes classification feature consistency loss, and the second loss is obtained based on the target face image and the current reconstructed face image, including: Using a pre-trained image classification network to classify the target face image, and obtain first feature data output by a target network layer of the image classification network; and Using the image classification network to classify the current reconstructed face image, and obtain the second feature data output by the target network layer; Obtaining the classification feature consistency loss based on the first feature data and the second feature data; The image classification network is trained using the target face image and the current reconstructed face image.

19. The method for generating a three-dimensional face model according to any one of claims 15 to 18, characterized in that: Performing texture feature restoration on a target face image including a target face to obtain initial texture feature data of the target face includes: Using a pre-trained second target neural network to restore texture features of the target face image, to obtain initial texture feature data of the target face; Wherein, the second target neural network is trained by using the first target neural network in any one of claims 11 to 15.

20. The method for generating a face model according to claim 19, characterized in that: The second target neural network is trained in the following manner: Using the first target neural network, processing the second sample normal vector image and the second sample texture feature data of the sample three-dimensional face model to obtain a sample face image of the sample three-dimensional face model; Processing the sample face image using a second neural network to be trained to obtain predicted texture feature data corresponding to the sample face image; Determining a third loss based on the predicted texture feature data and the second sample texture feature data; Based on the third loss, the second neural network to be trained is trained to obtain a second target neural network.

21. A facial image generating device, characterized in that: include: A first acquisition module is used to acquire a normal vector image and texture feature data; wherein the pixel value of each pixel point in the normal vector image represents the value of the normal vector of the model vertex corresponding to the pixel point in the three-dimensional face model corresponding to the normal vector image; A first processing module, used for performing multi-level data fusion processing on the normal vector image and the texture feature data to obtain a reconstructed face image; The first processing module, when performing multi-level data fusion processing on the normal vector image and the texture feature data to obtain a reconstructed face image, is used to: perform feature transformation on the texture feature data to obtain transformed texture feature data corresponding to the multi-level data fusion processing; For each level of data fusion processing in the multi-level data fusion processing, performing a first feature fusion on the transformed texture feature data corresponding to the level of data fusion processing and the input feature map corresponding to the level of data fusion processing to obtain an intermediate feature map of the level of data fusion processing; Performing a second feature fusion on the intermediate feature map of the data fusion processing at this level and the normal vector image to obtain a result feature map corresponding to the data fusion processing at this level; Based on the corresponding result feature map of the last level of data fusion processing, the reconstructed face image is obtained.

22. A 3D face model generation device, characterized in that: include: A second acquisition module is used to perform three-dimensional face reconstruction and texture feature recovery on a target face image including a target face, and obtain an initial normal vector image of the target face and initial texture feature data of the target face; A second processing module, configured to obtain a three-dimensional face model of the target face based on the initial normal vector image and the initial texture feature data; The second processing module, when obtaining the three-dimensional face model of the target face based on the initial normal vector image and the initial texture feature data, is used to: Using the initial normal vector image as the current normal vector image and the initial texture feature data as the current texture feature data, Generate a current reconstructed face image based on the current normal vector image and the current texture feature data; Based on the target face image and the current reconstructed face image, adjusting the current normal vector image to obtain a target normal vector image of the target face; Based on the target normal vector image, generating a three-dimensional face model of the target face; The second processing module, when generating a reconstructed face image based on the current normal vector image and the texture feature data, is used to: Using the current normal vector image as the normal vector image and the current texture feature data as the texture feature data, and using the face image generation method according to any one of claims 1 to 14 to generate a reconstructed face image; The reconstructed facial image is used as the current reconstructed facial image generated based on the current normal vector image and the current reconstructed facial image.

23. An electronic device, characterized in that: include: A processor and a memory, wherein the memory stores machine-readable instructions executable by the processor, and the processor is used to execute the machine-readable instructions stored in the memory. When the machine-readable instructions are executed by the processor, the processor executes the facial image generation method as described in any one of claims 1 to 14, or executes the three-dimensional facial model generation method as described in any one of claims 15 to 20.

24. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program. When the computer program is executed by an electronic device, the electronic device executes the facial image generation method as described in any one of claims 1 to 14, or executes the three-dimensional facial model generation method as described in any one of claims 15 to 20.

Images