Three-dimensional virtual character generation method, electronic equipment, storage medium and program product

Abstract

The embodiment of the invention provides a three-dimensional virtual character generation method, electronic equipment, a storage medium and a program product. The method comprises the following steps: acquiring a generation condition of a three-dimensional virtual character; encoding the generation condition to obtain a corresponding generation condition feature; processing the generation condition feature by using a three-dimensional role generation model to obtain a second face texture feature and a second face structure feature; wherein the three-dimensional role generation model is obtained by training based on a plurality of face image groups and image information of each group of face images, the face image groups comprise images of the same face under different illumination and different postures, and the image information comprises face attribute description and identity identification; and inputting the second face texture feature and the second face structure feature into a three-dimensional rendering engine to obtain a target three-dimensional virtual character output by the three-dimensional rendering engine. The method is used for achieving the effect of quickly generating a high-quality character head portrait of a virtual character based on text and / or image prompts.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a three-dimensional virtual character generation method, electronic equipment, storage medium and program product. Background Art

[0002] In the fields of virtual reality, augmented reality, education, art creation, games, movies, and human-computer interaction, three-dimensional virtual characters can be created based on text and / or images.

[0003] In related technologies, a 3D model of a 3D virtual character can be created by using the following methods: 3D Generative Adversarial Network (3D GAN), Variational Autoencoder (VAE), and Generative Adversarial Network (GAN) combined with a pose estimation algorithm. Compared with other parts of the body, the head of a virtual character requires more detailed 3D structure data and facial texture data during the creation process.

[0004] Due to the limited amount of relevant data and the difficulty in ensuring the generalization ability of the neural network model used in the existing technology, the current technology for generating virtual character portraits has the problem that the generated character portraits are too smooth, it is difficult to restore the high-frequency details of the texture, and the generated results are not realistic enough. Summary of the invention

[0005] The embodiments of the present application provide a three-dimensional virtual character generation method, electronic device, storage medium and program product, which are used to achieve the effect of quickly generating high-quality virtual character portraits based on text and / or image prompts.

[0006] In a first aspect, an embodiment of the present application provides a method for generating a three-dimensional virtual character, comprising:

[0007] Acquire generation conditions of a three-dimensional virtual character, the generation conditions including text description information and / or image description information;

[0008] The generation condition is encoded to obtain corresponding generation condition features; the generation condition features include a first face structure feature, a first face texture feature and a face occlusion feature;

[0009] The generation condition features are processed using a three-dimensional character generation model to obtain a second face texture feature and a second face structure feature; wherein the three-dimensional character generation model is trained based on multiple face image groups and image information of each face image group, the face image group includes images of the same face under different illuminations and different postures, and the image information includes facial attribute descriptions and identity identifiers;

[0010] The second facial texture feature and the second facial structure feature are input into a three-dimensional rendering engine to obtain a target three-dimensional virtual character output by the three-dimensional rendering engine.

[0011] In a possible implementation, the generation condition feature is processed using the three-dimensional character generation model to obtain the second face texture feature and the second face structure feature, including:

[0012] Performing texture reconstruction processing on the first facial texture feature and / or the facial occlusion feature to obtain a second facial texture feature corresponding to the generation condition; wherein the first facial texture feature is obtained by processing the texture text in the text description information through a text encoder; and the facial occlusion feature is obtained by expanding the image description information;

[0013] The first facial structure feature is geometrically reconstructed to obtain a second facial structure feature corresponding to the generation condition; wherein the first facial structure feature is obtained by processing the geometric text in the text description through a text encoder; or, when the generation condition only includes image description information, the second facial structure feature is estimated using a pre-trained face reconstruction model.

[0014] In a possible implementation, performing texture reconstruction processing on the first facial texture feature and / or the facial occlusion feature to obtain a second facial texture feature corresponding to the generation condition includes:

[0015] Using a texture diffusion model to process the first face texture feature and / or the face occlusion feature, to obtain a third face texture feature corresponding to the generation condition; wherein the texture diffusion model is trained based on multiple face image groups and image information of each face image group, the face image group includes images of the same face under different illuminations and different postures, and the image information includes facial attribute description and identity identification;

[0016] The resolution of the third facial texture feature is adjusted using a pre-trained super-resolution network to obtain a texture that meets the target resolution as the second facial texture feature.

[0017] In a possible implementation, the first facial texture feature and / or the facial occlusion feature is processed using a texture diffusion model to obtain a third facial texture feature corresponding to the generation condition, including:

[0018] Use the pre-trained encoder to encode the face occlusion features to obtain the incomplete latent variables corresponding to the face occlusion features;

[0019] Adding the first face texture feature to the texture diffusion model by means of cross attention as the first guiding condition of the texture diffusion model;

[0020] The given first Gaussian noise is used as a noise latent variable, concatenated with the incomplete latent variable, and input into the texture diffusion model, so that the texture diffusion model predicts and denoises the noise latent variable based on the first guided condition and the incomplete latent variable, and outputs the texture latent variable corresponding to the first guided condition;

[0021] The texture latent variables are decoded using a pre-trained decoder to obtain the third person face texture features corresponding to the generation conditions.

[0022] In a possible implementation, geometrically reconstructing the first facial structure feature to obtain the second facial structure feature corresponding to the generation condition includes:

[0023] The first facial structure feature is added to the geometric diffusion model by cross-attention as the second guiding condition of the geometric diffusion model; wherein the geometric diffusion model is trained based on multiple facial image groups and image information of each facial image group, the facial image group includes images of the same face under different illuminations and different postures, and the image information includes facial attribute description and identity identification;

[0024] Inputting the given second Gaussian noise into the geometric diffusion model, so that the geometric diffusion model predicts and denoises the second Gaussian noise based on the second guiding condition, and obtains the identity coefficient corresponding to the first facial structure feature;

[0025] The identity coefficients are subjected to basis linear weighting processing to obtain the second facial structure features corresponding to the generation conditions.

[0026] In a possible implementation, a method for constructing multiple facial image groups and image information of each facial image group includes:

[0027] Acquire multiple face images;

[0028] For each face image, delighting and relighting are performed to obtain a delighted multi-view face image and a relighted face image;

[0029] Based on the de-illuminated multi-view face image and the re-illuminated face image, images under different illuminations and different postures and albedo texture UV maps corresponding to the face image are obtained, and a correspondence is established between the images under different illuminations and different postures and the albedo texture UV maps and the face image to obtain a face image group;

[0030] A frontal view face image is selected from the de-illuminated multi-view face images, and identity coefficient estimation and facial attribute description processing are performed on the frontal view face image to obtain a facial attribute description corresponding to the face image and an identity identifier corresponding to the face image as image information of the face image;

[0031] For each face image, a correspondence is established between the face image group and the image information to obtain a plurality of face image groups and the image information of each face image group.

[0032] In a second aspect, an embodiment of the present application provides a three-dimensional virtual character generation device, comprising:

[0033] An acquisition module, used to acquire generation conditions of a three-dimensional virtual character, wherein the generation conditions include text description information and / or image description information;

[0034] The encoding module is used to encode the generation condition to obtain the corresponding generation condition feature; the generation condition feature includes the first face structure feature, the first face texture feature and the face occlusion feature;

[0035] a condition processing module, used to process the generated condition features using a three-dimensional character generation model to obtain a second face texture feature and a second face structure feature; wherein the three-dimensional character generation model is trained based on a plurality of face image groups and image information of each face image group, the face image group includes images of the same face under different illuminations and different postures, and the image information includes facial attribute descriptions and identity identifiers;

[0036] The generation module is used to input the second face texture feature and the second face structure feature into a three-dimensional rendering engine to obtain a target three-dimensional virtual character output by the three-dimensional rendering engine.

[0037] In a third aspect, an embodiment of the present application provides an electronic device, including: a memory, a processor;

[0038] The memory stores computer-executable instructions;

[0039] The processor executes the computer-executable instructions stored in the memory, so that the processor executes the above first aspect and / or various possible implementations of the first aspect.

[0040] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, in which computer-executable instructions are stored. When the computer-executable instructions are executed by a processor, they are used to implement the first aspect above and / or various possible implementations of the first aspect.

[0041] In a fifth aspect, an embodiment of the present application provides a computer program product, including a computer program, which, when executed by a processor, implements the above first aspect and / or various possible implementation methods of the first aspect.

[0042] The three-dimensional virtual character generation method, electronic device, storage medium and program product provided in the embodiments of the present application obtain the generation conditions of the three-dimensional virtual character, encode the generation conditions and extract facial structure features, and use the three-dimensional character generation model trained with a texture data set for processing, and render through a three-dimensional rendering engine to obtain the three-dimensional virtual character corresponding to the generation conditions, thereby achieving the effect of quickly generating high-quality virtual character portraits based on text and / or image prompts. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0044] Figure 1 Schematic diagram of the process of generating a three-dimensional virtual character provided in this application Figure 1 ;

[0045] Figure 2 Schematic diagram of the process of generating a three-dimensional virtual character provided in this application Figure 2 ;

[0046] Figure 3 A schematic diagram of the process of generating a three-dimensional virtual character provided Figure 3 ;

[0047] Figure 4 A schematic diagram of the process of generating a three-dimensional virtual character provided Figure 4 ;

[0048] Figure 5 A schematic diagram of a process for constructing a data set in one embodiment of the present application;

[0049] Figure 6 A schematic diagram of the structure of a three-dimensional character generation model used in one embodiment of the present application;

[0050] Figure 7 A schematic diagram of the structure of the three-dimensional virtual character generation device provided in this application;

[0051] Figure 8 A schematic diagram of the structure of the electronic device provided in this application.

[0052] The above drawings have shown clear embodiments of the present application, which will be described in more detail later. These drawings and text descriptions are not intended to limit the scope of the present application in any way, but to illustrate the concept of the present application to those skilled in the art by referring to specific embodiments. DETAILED DESCRIPTION

[0053] Exemplary embodiments will be described in detail herein, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, the same numbers in different drawings represent the same or similar elements unless otherwise indicated. The implementations described in the following exemplary embodiments do not represent all implementations consistent with the present application. Instead, they are merely examples of devices and methods consistent with some aspects of the present application as detailed in the appended claims.

[0054] First, the terms involved in this application are explained:

[0055] Wild images: Face images obtained from the real world without any specific screening or editing. These images may come from various sources such as photos, videos, surveillance videos, social media, etc., and they represent face image data in real environments. In this application, in order to simulate wild images, the re-illuminated images with complex lighting and posture obtained by re-illuminating and adjusting the posture of specific face images can also be called wild images.

[0056] 3D geometric identity coefficients: also known as 3DMM (Three-Dimensional Morphable Model) parameters, are used to describe and represent the three-dimensional face shape and appearance. These parameters can be interpreted as a compact representation for generating three-dimensional face models of different individuals. In this application, the facial structural features are the feature vector representations of the 3D geometric identity coefficients.

[0057] Texture UV Mapping: In the field of computer graphics and 3D modeling, UV texture mapping is a technique for applying a 2D image (texture) to the surface of a 3D model. UV texture coordinates are a system for representing texture mapping, which maps points on a 3D model to points on a 2D texture image. Each point on the surface of a 3D model can be defined with corresponding UV coordinates, which define its position in the texture image. These coordinates map the 2D image to the surface of the 3D model through UV mapping, ultimately forming a texture effect.

[0058] High-quality facial texture UV map: refers to a facial albedo texture UV map that is evenly illuminated and free of unnecessary occlusions such as hair and glasses. This texture UV map can be used as a facial data asset for training related neural network models under different lighting conditions.

[0059] Currently, in the 3D virtual character generation technology, due to the lack of large datasets containing detailed facial description text and high-quality 3D information (texture UV maps and shapes), the 3D virtual character generation method is unable to generate high-quality face models (also called head models or facial models) based on complex prompts.

[0060] In view of the above problems, the present application provides a method for generating a three-dimensional virtual character, firstly obtaining the generation conditions of the three-dimensional virtual character; encoding the generation conditions to obtain corresponding generation condition features; the generation condition features include a first face structure feature, a first face texture feature and a face occlusion feature; using a three-dimensional character generation model to process the generation condition features to obtain a second face texture feature and a second face structure feature; wherein the three-dimensional character generation model is trained based on multiple face image groups and image information of each face image group, the face image group includes images of the same face under different illumination and different postures, and the image information includes facial attribute description and identity identification; the second face texture feature and the second face structure feature are input into a three-dimensional rendering engine to obtain a target three-dimensional virtual character output by the three-dimensional rendering engine. The method is used to achieve the effect of quickly generating high-quality virtual character avatars based on text and / or image prompts.

[0061] The technical solution of the present application and how the technical solution of the present application solves the above-mentioned technical problems are described in detail below with specific embodiments. The following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be repeated in some embodiments. The embodiments of the present application will be described below in conjunction with the accompanying drawings.

[0062] Figure 1 Schematic diagram of the process of generating a three-dimensional virtual character provided in this application Figure 1 ,like Figure 1 As shown, the method includes:

[0063] S101: Acquire generation conditions of a three-dimensional virtual character, where the generation conditions include text description information and / or image description information.

[0064] S102, encoding the generation conditions to obtain corresponding generation condition features; the generation condition features include a first face structure feature, a first face texture feature and a face occlusion feature.

[0065] Specifically, for text description information, a text encoder may be used for encoding, and for image description information, a VAE encoder may be used for encoding.

[0066] Among them, the facial structure feature can be extracted from the text description information and / or the image description information, and is used to indicate the structural information such as the geometric identity coefficient of the face of the three-dimensional virtual character to be generated. The first facial texture feature can be extracted from the text description information and / or the image description information, and is used to indicate the texture information of the face to be generated. The facial occlusion feature can be extracted from the image description information, and is used to indicate the occlusion information of the face to be generated. Exemplarily, the facial occlusion feature can also be called the incomplete texture feature, which is used to characterize the part information of the face that lacks texture.

[0067] S103. Use a three-dimensional character generation model to process the generation condition features to obtain a second facial texture feature and a second facial structure feature; wherein the three-dimensional character generation model is trained based on multiple facial image groups and image information of each group of facial images, the facial image group includes images of the same face under different lighting and different postures, and the image information includes facial attribute descriptions and identity identification.

[0068] Among them, the three-dimensional character generation model can include a face structure reconstruction network and a face texture reconstruction network, which are used to process face structure features and texture features respectively.

[0069] Specifically, a face texture reconstruction network can be used to perform texture reconstruction on the first face texture feature and / or face occlusion feature to obtain a second face texture feature corresponding to the generation condition; wherein the first face texture feature is obtained by processing the texture text in the text description information through a text encoder; and the face occlusion feature is obtained by expanding the image description information.

[0070] Exemplarily, the face texture reconstruction network can be a texture diffusion model. The first face texture feature and / or face occlusion feature is processed using the texture diffusion model to obtain a third face texture feature corresponding to the generation condition. The texture diffusion model is trained based on multiple face image groups and image information of each group of face images. The face image group includes images of the same face under different illumination and different postures, and the image information includes facial attribute description and identity identification. The identity identification can be a parameter such as an identity coefficient (also called a geometric coefficient) of a face image, which is used to characterize the 3D geometry of the face image. Furthermore, the resolution of the third face texture feature can be adjusted using a pre-trained super-resolution network to obtain a texture that meets the target resolution as the second face texture feature.

[0071] Optionally, a facial structure reconstruction network can be used to perform geometric reconstruction on the first facial structure feature to obtain a second facial structure feature corresponding to the generation condition; wherein the first facial structure feature is the geometric text in the text description processed by a text encoder; or, when the generation condition only includes image description information, a pre-trained face reconstruction model can be used to estimate the second facial structure feature.

[0072] S104: Input the second facial texture feature and the second facial structure feature into a three-dimensional rendering engine to obtain a target three-dimensional virtual character output by the three-dimensional rendering engine.

[0073] The 3D rendering engine can be computer software or a set of algorithms for generating 3D images. The input of the 3D rendering engine can be a feature vector or other data representing a 3D model obtained by processing a neural network model, and the output can be a 2D image of the 3D model corresponding to the vector. In the field of 3D virtual character generation, the output of the 3D rendering engine is the target 3D virtual character.

[0074] Specifically, the first facial structure feature may be a feature vector encoded by facial geometric structure information extracted from the text, and the second facial structure feature may be a latent space vector obtained by denoising the Gaussian noise with the first facial structure feature as a guiding condition, and then processing the latent space vector using a deep learning network model to obtain geometric structure data of a three-dimensional facial model; exemplarily, the second facial structure feature may be a feature vector.

[0075] The first facial texture feature may be a feature vector encoded from facial texture information extracted from text description information and / or image description information, and the second facial texture feature may be a latent space vector obtained by using the first facial texture feature as a guiding condition, denoising random Gaussian noise with a texture diffusion model, and then decoding the texture data of a three-dimensional facial model through a decoder; exemplarily, the second facial texture feature may be a feature vector.

[0076] The three-dimensional virtual character generation method provided in the embodiment of the present application obtains the generation conditions of the three-dimensional virtual character, encodes the generation conditions and extracts facial structure features, and uses a three-dimensional character generation model trained with a texture data set for processing, and renders through a three-dimensional rendering engine to obtain a means of obtaining a three-dimensional virtual character corresponding to the generation conditions, thereby achieving the effect of quickly generating high-quality virtual character portraits based on text and / or image prompts.

[0077] In one possible implementation, Figure 2 Schematic diagram of the process of generating a three-dimensional virtual character provided in this application Figure 2 ,like Figure 2 As shown, a method for constructing multiple face image groups and image information of each face image group includes the following steps:

[0078] Step 201: Acquire multiple face images.

[0079] The face image may be a multi-view de-illumination face image, wherein the front view provides frontal facial features, and the left view and the right view provide side facial features.

[0080] Step 202: For each face image, perform delighting and relighting to obtain a delighted multi-view face image and a relighted face image in a complex posture.

[0081] Specifically, face processing can be processed using the deep learning network of NeRFFaceLighting. NeRFFaceLighting is a 3D-aware adversarial generative network (GAN) that can generate de-lighted face images at specified poses and specified spherical harmonic coefficients, as well as re-illuminated face images at specified spherical harmonic coefficients, while maintaining excellent identity and geometric consistency. Specifically, Gaussian noise or existing images can be mapped or projected into the latent space using a mapping or pre-trained projection network. Then, NeRFFaceLighting is used to generate forward de-lighted face images, as well as multi-view images with deflection angles of 35 degrees to the left and right. Finally, poses and spherical harmonic coefficients can be randomly sampled from the pose and spherical harmonic distributions of the FFHQ dataset, and NeRFFaceLighting can be guided to generate re-illuminated images with complex poses and lighting to simulate wild images.

[0082] Step 203: Based on the de-illuminated multi-view face image and the re-illuminated face image, images under different illuminations and different postures, and albedo texture UV maps corresponding to the face images are obtained, and a correspondence is established between the images under different illuminations and different postures, and the albedo texture UV maps and the face images to obtain a face image group.

[0083] Specifically, multiple views can be used to illuminate the face image I f ,I l ,I r To synthesize a high-quality facial albedo texture UV map, first separate the f ,I l ,I r Extract the corresponding incomplete UV texture T f ,T l ,T r , called "unfolding". We use the Deep3D model pre-trained on HIFI3D++ to estimate the 3DMM coefficients of each face image and calculate the 3D vertex coordinates. Subsequently, the face image is projected onto 3D vertices and unfolded into a facial UV texture by defining a mapping between vertices and UV mapping coordinates. Furthermore, the facial parsing model can also be used to extract the mask of the facial area and map it into the UV space to filter out non-facial areas in the UV texture. Similar to FFHQ-UV, the predefined visibility masks (front and side face UV masks), detected defect masks (eyes, nose and mouth), and template texture T^ are used to obtain T through linear blending, color correction, and texture completion. f ,T l ,T rThe final albedo texture UV map.

[0084] Step 204: select a frontal view face image from the de-illuminated multi-view face images, perform identity coefficient estimation and facial attribute description processing on the frontal view face image, and obtain a facial attribute description corresponding to the face image and an identity identifier corresponding to the face image as image information of the face image.

[0085] Specifically, Deep3D can be used to extract identity coefficients and facial attribute descriptions from facial images. Since NeRFFacelighting is able to preserve identity, we only operate on de-illuminated frontal facial images. First, we estimate the facial identity coefficient α∈R532 using a Deep3D model pre-trained on HIFI3D++. With the booming development of large visual language models, which effectively establish a matching relationship between images and natural language, we no longer use expensive manual annotations. Instead, we use carefully designed instructions to interact with the GPT-4V(vision) API3 to automatically generate detailed and diverse basic, texture, and geometric descriptions with an average length of 200 tokens.

[0086] Step 205: For each face image, a correspondence is established between the face image group and the image information to obtain a plurality of face image groups and the image information of each face image group.

[0087] In some examples, the face image group and the image information of each group of face images together constitute a face texture UV map dataset for subsequent model training.

[0088] The method for generating a three-dimensional virtual character provided in the embodiment of the present application obtains a low-cost and reproducible large-scale dataset by constructing a large-scale human face texture UV map dataset, which contains hundreds of thousands of pairs of text descriptions and 3D facial information. The dataset contains four modalities: a detailed description of facial attributes, a wild map with complex lighting and posture, a high-quality albedo texture UV map, and a 3D geometric identity coefficient corresponding to facial features. The creation process of the dataset is low-cost and reproducible.

[0089] Based on this data set, the three-dimensional virtual character generation method provided in the embodiment of the present application can solve the problem of missing existing text or images and corresponding 3D avatar data (geometry and texture), and on this basis complete the 3D avatar generation based on text or single image prompts.

[0090] In one possible embodiment, Figure 3 A flowchart of a method for generating a three-dimensional virtual character is provided, such as Figure 3As shown, using the texture diffusion model to process the first face texture feature and / or the face occlusion feature to obtain the third face texture feature corresponding to the generation condition may include the following steps:

[0091] Step 301: Use a pre-trained encoder to encode the face occlusion feature to obtain an incomplete latent variable corresponding to the face occlusion feature.

[0092] The encoder may be a VAE encoder, and the face occlusion feature may be a defective texture corresponding to the face occlusion information in the image description information. Using the pre-trained VAE encoder, the defective texture corresponding to the image description information (also referred to as image prompt) may be encoded as a latent variable l.

[0093] Step 302: Add the first face texture feature to the texture diffusion 2 model by means of cross attention as the first guiding condition of the texture diffusion model.

[0094] Step 303: Use the given first Gaussian noise as a noise latent variable, concatenate it with the incomplete latent variable, and input it into the texture diffusion model, so that the texture diffusion model predicts and denoises the noise latent variable based on the first guided condition and the incomplete latent variable, and outputs the texture latent variable corresponding to the first guided condition.

[0095] Among them, the diffusion model is a type of generative model, the core idea of ​​which is to convert data into pure noise by gradually adding noise, and then gradually remove the noise to restore the original data through the inverse process. Diffusion models can be divided into many types according to the different types of processed data. The texture diffusion model (TDM) is an application of the diffusion model, which is used to generate and process texture images. Specifically, the first facial texture feature can be added to the texture diffusion model by cross-attention as the first guiding condition of the texture diffusion model. The given first Gaussian noise is used as the noise latent variable, spliced ​​with the incomplete latent variable, and input into the texture diffusion model. The texture diffusion model can predict and denoise the noise latent variable based on the first guiding condition and the incomplete latent variable, and output the texture latent variable corresponding to the first guiding condition.

[0096] In a specific application scenario, given Gaussian noise as the noise latent variable z T ∈R 3*64*64 , the incomplete latent variable ∈R 3*64*64 It is concatenated with the noise latent variable and then fed into the unet of TDM. The time step t (from T to 0) and the text embedding are given in turn to guide TDM to predict the size of the noise that should be removed through the form of cross attention, so as to achieve the next step from the noise latent variable z t-1Each time the noise is predicted, the incomplete latent variable is concatenated with the current noise latent variable until the texture latent variable z0 is obtained.

[0097] Step 304: Use the pre-trained decoder to decode the texture latent variable to obtain the third face texture feature corresponding to the generation condition.

[0098] Specifically, z0 can be passed through the VAE decoder to obtain the texture corresponding to the current text prompt or face image prompt, that is, the third face texture feature. Furthermore, the resolution is enlarged to 2K through the pre-trained super-resolution network to obtain the second face texture feature.

[0099] In a specific application scenario, the second face texture feature can be a UV texture map, and a high-quality UV texture map as defined in this application. The face texture description and the wild image are both guiding conditions for the high-quality UV texture map, and the wild image is used to extract the incomplete texture.

[0100] The three-dimensional virtual character generation method provided in the embodiment of the present application uses a texture diffusion model to process texture information in text and / or image prompts, thereby achieving a realistic texture effect for the generated three-dimensional virtual character.

[0101] In one possible implementation, Figure 4 A schematic diagram of the process of generating a three-dimensional virtual character provided Figure 4 ,like Figure 4 As shown, performing geometric reconstruction processing on the first facial structure feature to obtain the second facial structure feature corresponding to the generation condition may include the following steps:

[0102] Step 401: Add the first facial structure feature to the geometric diffusion model by means of cross attention as the second guiding condition of the geometric diffusion model.

[0103] Among them, the geometric diffusion model (GDM) is an application of the diffusion model, which is mainly used to generate and process the geometric structure data of three-dimensional models and three-dimensional images. Specifically, the first face structure feature can be added to the geometric diffusion model by cross-attention as a guiding condition of the geometric diffusion model. GDM generates geometric coefficients through a structural descriptive language, thereby weighting the shape basis in the 3DMM model to obtain geometry. Exemplarily, the structural descriptive language input by GDM can be the first face structure feature.

[0104] Step 402: Input the given second Gaussian noise into the geometric diffusion model, so that the geometric diffusion model performs predictive denoising on the second Gaussian noise based on the second guiding condition to obtain the identity coefficient corresponding to the first facial structure feature.

[0105] The given Gaussian noise is used as a noise latent variable and input into the geometric diffusion model. The geometric diffusion model can predict and denoise the noise latent variable based on the second guiding condition, and output the corresponding geometric latent variable, that is, the identity coefficient corresponding to the first facial structure feature.

[0106] Step 403: Perform basis linear weighting processing on the identity coefficient to obtain a second facial structure feature corresponding to the generation condition.

[0107] For example, the identity basis linear weighting based on HIFI3D++ can be used to obtain the corresponding face geometry, that is, the second face structure feature.

[0108] The three-dimensional virtual character generation method provided in the embodiment of the present application obtains geometric structure feature data of the three-dimensional virtual character from the generation conditions by using a geometric diffusion model, thereby improving the realism of the generated three-dimensional virtual character.

[0109] In order to further explain the solution of the embodiment of the present application, a specific example is used below to illustrate. In a specific application scenario, the three-dimensional virtual character generation method provided in the embodiment of the present application can be divided into three steps: building a data set, model training, and character generation.

[0110] Figure 5 A schematic diagram of a process for constructing a data set in one embodiment of the present application is shown in FIG. Figure 5 As shown, the embodiment of the present application first proposes a cost-effective and replicable data generation pipeline to construct a large-scale face texture UV map dataset. The relightable 3D-aware GAN model NeRFFacelight and texture correction and supplementation from FFHQ-UV are used to create more than 100,000 high-quality 1K resolution albedo rate texture UV maps. Each UV map is paired with a 3DMM identity coefficient, fine texture and geometric texture description (provided by GPT-4vision), and five re-lit and pose-changed facial images. Specifically, the dataset construction can be divided into three steps: the first two steps explain how to generate high-quality texture UV maps, 4mapping3DMM coefficient estimation models, identity coefficients, projections, and corresponding wild face images (re-lit images with complex lighting and postures). The third step describes how to generate the corresponding identity coefficients (3D geometry) and detailed text descriptions of facial features.

[0111] In the first step: Figure 5As shown, high-quality UV textures are extracted from multi-view de-illuminated face images. Among them, the front view provides frontal facial features, and the left and right views provide side facial features. At the same time, it aims to generate multi-view de-illuminated face images {R1, ..., R5} of the same identity with random lighting and poses. NeRFFaceLighting is used to achieve these goals. NeRFFaceLighting is a 3D-aware adversarial generative network (GAN) that can generate de-illuminated face images at specified poses and specified spherical harmonic coefficients, as well as re-illuminated face images at specified spherical harmonic coefficients, while maintaining excellent identity and geometric consistency. Specifically, Gaussian noise or existing images are mapped or projected into the latent space using a mapping or pre-trained projection network. Then, NeRFFaceLighting is used to generate de-hazed face images in the frontal control position, as well as deflection angles of 35 degrees to the left and right. Finally, pose and spherical harmonic coefficients are randomly sampled from their distributions in the FFHQ dataset and guided to generate re-lighted images with complex poses and lightings to simulate images in the wild.

[0112] In the second step: use multiple views to illuminate the face image I f ,I l ,I r To synthesize high-quality facial albedo texture UV maps. Figure 2 As shown, first, from I f ,I l ,I r Extract the corresponding incomplete UV texture T f ,T l ,T r , called "unfolding". A Deep3D model based on HIFI3D++ pre-training is used to estimate the 3DMM coefficients of each face image and calculate the 3D vertex coordinates. Subsequently, the face image is projected onto 3D vertices and unfolded into a facial UV texture by defining a mapping between vertices and UV mapping coordinates. The facial parsing model 2 is also used to extract the mask of the facial area and map it into the UV space to filter out non-facial areas in the UV texture. Similar to FFHQ-UV, T is obtained by linear blending, color correction, and texture completion using predefined visibility masks (front and side face UV masks), detected defect masks (eyes, nose, and mouth), and template texture T^. f ,T l ,T r The final albedo texture UV map.

[0113] In the third step: In this step, Deep3D is used to extract the identity coefficient and facial attribute description of the facial image. Since NeRFFacelighting is able to preserve identity, it is only operated on the de-illuminated front face image. First, the facial identity coefficient α∈R532 is estimated using the Deep3D model pre-trained on HIFI3D++. With the booming development of large-scale visual language models, these models effectively establish a matching relationship between images and natural language, and no longer use expensive manual annotations. Instead, carefully designed instructions are used to interact with the GPT-4V(vision) API3 to automatically generate detailed and diverse basic, texture, and geometric descriptions with an average length of 200 tokens.

[0114] Based on the above data set, the texture generation model TDM and the geometry generation model GDM of PromptAvatar (i.e., the three-dimensional character generation model of the embodiment of the present application) can be trained. The structure of the PromptAvatar model is as follows: Figure 6 shown.

[0115] The training of TDM consists of two parts: first, VAE, which consists of an Encoderε and Composition, responsible for compressing the texture UV map x into the feature latent space; subsequently, the weights of the VAE encoder are fixed, and the diffusion model (DM) is trained based on the image and text cues in the latent space to establish a mapping relationship between Gaussians. In order to make full use of the valuable features provided by facial image cues in unsupervised learning, an incomplete UV texture is extracted from the image using the "unwarp" operation. The incomplete texture UV map is encoded using ε to obtain l∈R4×64×64, and is connected to the noisy latent variable zt at different time steps. Injecting information about incomplete texture into the diffusion model helps maintain the identity consistency between the generated facial texture and the input image. As for the text cues, it is embedded using the text encoder of CLIP, and the text embedding is sent to the cross-attention layer of DM. Both the image cue Ip and the text cue yp are used as conditions to train the denoising text model ∈tex of TDM through the following loss. Finally, LDM (Latent Diffusion Models) is responsible for performing the diffusion model in the feature latent space under the guidance of y, that is, the noise prediction network ∈ θ y is trained to generate the corresponding texture UV map from the noise. Where y is injected into the denoising process through the cross-attention layer.

[0116] During the training process, the first stage first trains the VAE alone by optimizing the following loss function:

[0117]

[0118] in, They are KL divergence regularization loss, adversarial loss and LPIPS perception loss. Then LCM (Latent Consistency Model) and LDM are jointly trained and optimized through the following loss:

[0119]

[0120] where t is a time step uniformly sampled from {1, ..., T}, and the potential zt is obtained based on t and z0(21; 55). During training, p drop = 0.05 probability of randomly discarding image prompts, text prompts, or both, so that our generative model can adapt to non-prompted, single-prompted, and multi-prompted inputs. By extracting facial information from text or image prompts, our TDM can generate realistic texture UV maps with details to help 3D avatar generation.

[0121] The Geometric Diffusion Model (GDM) can generate neutral facial geometry based on textual cues. In 3DMM, identity coefficients refer to a set of parameters used to describe the shape of the face. By adjusting these coefficients, the characteristics of the model, such as facial contours, eye shape, lip size, etc., can be changed. Identity coefficients play an important role in modeling and controlling the mesh of the face within the 3DMM framework. Figure 6 As shown, GDM is trained in the identity coefficient space. Given the paired data of geometric text prompts Pgeo and identity coefficients α in the dataset, Pgeo is encoded into text embeddings using CLIP's text encoder, enabling GDM to predict the noise applied to the identity coefficient at time step t under the guidance of the embedding. GDM is similar in structure to the DM of TDM, but it is designed as a 1D UNet and maps the text and identity coefficients together through a cross-attention mechanism. Similar to the training of TDM, GDM is trained with the following loss:

[0122]

[0123] where h t is the identity coefficient after applying noise at time step t.

[0124] In a specific example, the parameters for model training are set as follows:

[0125] For TDM, during training, the weights were initialized using the SD-2.14 model and the weights of the first convolutional layer of UNet were copied to expand the input dimension to 8. The albedo texture UV map and the incomplete texture extracted from the image prompt were resized to 512×512 resolution to fit the VAE. During training, PNDMScheduler was adopted and the time step T was set to 1000. During inference, the sampling step was set to 100, with w=6 in Equation 2. After TDM samples the latent embedding through UNet based on text or image prompts, the resulting 512×512 resolution texture is fed into D. This low-resolution texture is then passed to a pre-trained super-resolution network (69) to obtain the final 2K resolution texture UV map. GDM was trained from scratch, with w=1.5 and a time step of 200 during inference. The VAE for TDM was fine-tuned using the Adam optimizer (28) and trained for two days with β1 = 0.5, β2 = 0.9, and a learning rate η1 = 4.5e -6 Both TDM’s Unet and GDM use the AdamW optimizer (41), where β1 = 0.9, β2 = 0.999, weight decay μ = 0.01, and learning rate η2 = 10 × 10 -5 , the former took seven days and the latter took two days. All training was performed on eight NVIDIA A800-SXM4-80GB GPUs, and inference was performed on a single GPU. During training, since CLIP's text encoder accepts a maximum of 77 tokens, the sentences are randomly combined and truncated before feeding the text prompts to the encoder to enhance the diversity of the text prompts.

[0126] The trained 3D character generation model above mainly processes the data in the following way in the process of generating a 3D virtual character (also called 3D avatar) based on text and / or image prompts:

[0127] Step 1, PromptAvatar supports three types of prompts to guide 3D avatar generation. The first is a single facial attribute text prompt. The basic attributes and texture attribute descriptions directly guide TDM to generate the corresponding facial texture, and the basic attributes and geometric attributes directly guide GDM to generate the corresponding geometry; the second is a wild face image (allowing difficult distribution of posture and illumination), which directly guides TDM to generate the corresponding face texture, and then uses the pre-trained face reconstruction model Deep3D trained based on the HIFI3D++ face deformable model to estimate the geometry of the face image; the third is a mixed input of facial attribute text and face image. First, a general text-based large model (such as Delle3, SDXL, etc.) can be used to generate the corresponding face image based on a given text description, and then the text and the face image jointly guide TDM to generate the corresponding texture, and then the above-mentioned Deep3D model is used to directly estimate the geometry from the face image.

[0128] Step 2, specifically for the generation of TDM, given a texture text prompt, first use the text encoder of the CLIP model to encode the text into a texture text embedding; given a face image prompt, use the "expand" operation to obtain the corresponding face incomplete texture: use the pre-trained Deep3D model to estimate the identity coefficient, expression coefficient, and posture parameter of the face in the face image, then perform linear weighting and posture parameter transformation on the face deformable model HIFI3D++ based on the identity coefficient and posture coefficient to obtain the vertex coordinates of the face's geometric mesh. Through rasterization, the two-dimensional coordinates of the mesh vertex corresponding to the face image can be found, and then the mesh vertex color can be obtained from the face image through bilinear interpolation. Then, using the mapping relationship between the predefined 3D vertex and UV coordinates, the mesh vertex color can be projected to the UV coordinate, thereby obtaining the incomplete texture corresponding to the face image as the image guidance condition, and finally use the pre-trained VAE encoder to encode the incomplete texture corresponding to the image prompt into the latent variable l. For the geometric text prompt of GDM, the text encoder of the CLIP model is also used to encode the text into a geometric text embedding.

[0129] Step 3, for TDM: Given Gaussian noise as the noise latent variable z T ∈R 3*64*64 , the incomplete latent variable ∈R 3 *64*64 It is concatenated with the noise latent variable and then fed into the unet of TDM. The time step t (from T to 0) and the text embedding are given in turn to guide TDM to predict the size of the noise that should be removed through the form of cross attention, so as to achieve the next step from the noise latent variable z t-1Calculation. Each time the noise is predicted, the incomplete latent variable is concatenated with the current noise latent variable until z0 is obtained. It should be noted that if only the text condition is given, the incomplete texture latent variable will be set to all 0s; if only the image condition is given, the texture text will be set to an empty string "".

[0130] Similarly, for GDM: we also use cross attention to embed the geometric texture to guide GDM from the one-dimensional Gaussian noise h T ∈R 532 Denoising is performed until h0 is obtained, which is directly used as the identity coefficient of the current geometric description.

[0131] Step 4, for TDM: pass z0 through the VAE decoder to obtain the texture corresponding to the current text prompt or face image prompt, and upscale it to 2K in the pre-trained super-resolution network. For GDM: linearly weight the predicted identity coefficients based on the identity basis of HIFI3D++ and the corresponding face geometry. Put the face texture and geometry into the existing 3D rendering engine as the final 3D avatar result.

[0132] Figure 7 A schematic diagram of the structure of the three-dimensional virtual character generation device provided in this application, such as Figure 7 As shown, the three-dimensional virtual character generation device 70 provided in this embodiment includes:

[0133] An acquisition module 701 is used to acquire generation conditions of a three-dimensional virtual character, where the generation conditions include text description information and / or image description information;

[0134] The encoding module 702 is used to encode the generation condition to obtain the corresponding generation condition feature; the generation condition feature includes the first face structure feature, the first face texture feature and the face occlusion feature;

[0135] The condition processing module 703 is used to process the generated condition features using the three-dimensional character generation model to obtain the second face texture features and the second face structure features; wherein the three-dimensional character generation model is trained based on multiple face image groups and image information of each face image group, the face image group includes images of the same face under different illuminations and different postures, and the image information includes facial attribute descriptions and identity identifiers;

[0136] The generation module 704 is used to input the second facial texture feature and the second facial structure feature into a three-dimensional rendering engine to obtain a target three-dimensional virtual character output by the three-dimensional rendering engine.

[0137] In a possible implementation, the condition processing module is further used to:

[0138] Performing texture reconstruction processing on the first facial texture feature and / or the facial occlusion feature to obtain a second facial texture feature corresponding to the generation condition; wherein the first facial texture feature is obtained by processing the texture text in the text description information through a text encoder; and the facial occlusion feature is obtained by expanding the image description information;

[0139] The first facial structure feature is geometrically reconstructed to obtain a second facial structure feature corresponding to the generation condition; wherein the first facial structure feature is obtained by processing the geometric text in the text description through a text encoder; or, when the generation condition only includes image description information, the second facial structure feature is estimated using a pre-trained face reconstruction model.

[0140] In a possible implementation, the condition processing module is further used to:

[0141] Using a texture diffusion model to process the first face texture feature and / or the face occlusion feature, to obtain a third face texture feature corresponding to the generation condition; wherein the texture diffusion model is trained based on multiple face image groups and image information of each face image group, the face image group includes images of the same face under different illuminations and different postures, and the image information includes facial attribute description and identity identification;

[0142] The resolution of the third facial texture feature is adjusted using a pre-trained super-resolution network to obtain a texture that meets the target resolution as the second facial texture feature.

[0143] In a possible implementation, the condition processing module is further used to:

[0144] Use the pre-trained encoder to encode the face occlusion features to obtain the incomplete latent variables corresponding to the face occlusion features;

[0145] Adding the first face texture feature to the texture diffusion model by means of cross attention as the first guiding condition of the texture diffusion model;

[0146] The given first Gaussian noise is used as a noise latent variable, concatenated with the incomplete latent variable, and input into the texture diffusion model, so that the texture diffusion model predicts and denoises the noise latent variable based on the first guided condition and the incomplete latent variable, and outputs the texture latent variable corresponding to the first guided condition;

[0147] The texture latent variables are decoded using a pre-trained decoder to obtain the third person face texture features corresponding to the generation conditions.

[0148] In a possible implementation, the condition processing module is further used to:

[0149] The first facial structure feature is added to the geometric diffusion model by cross-attention as the second guiding condition of the geometric diffusion model; wherein the geometric diffusion model is trained based on multiple facial image groups and image information of each facial image group, the facial image group includes images of the same face under different illuminations and different postures, and the image information includes facial attribute description and identity identification;

[0150] Inputting the given second Gaussian noise into the geometric diffusion model, so that the geometric diffusion model predicts and denoises the second Gaussian noise based on the second guiding condition, and obtains the identity coefficient corresponding to the first facial structure feature;

[0151] The identity coefficient is linearly weighted on the face shape basis of HIFI3D++ to obtain the second face structure feature corresponding to the generation condition.

[0152] In a possible implementation, the system further includes a data set construction module for:

[0153] Acquire multiple face images;

[0154] For each face image, delighting and relighting are performed to obtain a delighted multi-view face image and a relighted face image;

[0155] Based on the de-illuminated multi-view face image and the re-illuminated face image, images under different illuminations and different postures and albedo texture UV maps corresponding to the face image are obtained, and a correspondence is established between the images under different illuminations and different postures and the albedo texture UV maps and the face image to obtain a face image group;

[0156] A frontal view face image is selected from the de-illuminated multi-view face images, and identity coefficient estimation and facial attribute description processing are performed on the frontal view face image to obtain a facial attribute description corresponding to the face image and an identity identifier corresponding to the face image as image information of the face image;

[0157] For each face image, a correspondence is established between the face image group and the image information to obtain a plurality of face image groups and the image information of each face image group.

[0158] The three-dimensional virtual character generation device provided in this embodiment can execute the method provided in the above method embodiment. Its implementation principle and technical effect are similar, and this embodiment will not be described in detail here.

[0159] Figure 8 This is a schematic diagram of the structure of the electronic device provided in this application. Figure 8As shown, the electronic device 80 provided in this embodiment includes: at least one processor 801 and a memory 802. Optionally, the device 80 also includes a communication component 803. The processor 801, the memory 802 and the communication component 803 are connected via a bus 804.

[0160] In a specific implementation process, at least one processor 801 executes the computer-executable instructions stored in the memory 802, so that at least one processor 801 executes the above method.

[0161] The specific implementation process of the processor 801 can be found in the above method embodiment, and its implementation principle and technical effect are similar, so this embodiment will not be repeated here.

[0162] In the above embodiments, it should be understood that the processor may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), etc. A general-purpose processor may be a microprocessor or any conventional processor. The steps of the method disclosed in the invention may be directly implemented as being executed by a hardware processor, or may be executed by a combination of hardware and software modules in the processor.

[0163] The memory may include a high-speed memory (Random Access Memory, RAM), and may also include a non-volatile memory (Non-volatile Memory, NVM), such as at least one disk memory.

[0164] The bus may be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus, etc. The bus may be divided into an address bus, a data bus, a control bus, etc. For ease of representation, the bus in the drawings of the present application is not limited to only one bus or one type of bus.

[0165] The present application also provides a computer program product, including a computer program, which implements the above method when executed by a processor.

[0166] The present application also provides a computer-readable storage medium, in which computer-executable instructions are stored. When a processor executes the computer-executable instructions, the above method is implemented.

[0167] The above-mentioned readable storage medium can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk or optical disk. The readable storage medium can be any available medium that can be accessed by a general or special-purpose computer.

[0168] An exemplary readable storage medium is coupled to a processor so that the processor can read information from the readable storage medium and write information to the readable storage medium. Of course, the readable storage medium can also be a component of the processor. The processor and the readable storage medium can be located in an application specific integrated circuit (Application Specific Integrated Circuits, referred to as: ASIC). Of course, the processor and the readable storage medium can also exist in the device as discrete components.

[0169] The division of units is only a logical function division, and there may be other divisions in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interface, device or unit, which can be electrical, mechanical or other forms.

[0170] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0171] In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0172] If the function is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium, including several instructions for a computer device (which can be a personal computer, server, or network device, etc.) to perform all or part of the steps of the methods of each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc. Various media that can store program codes.

[0173] Those skilled in the art can understand that all or part of the steps of implementing the above-mentioned method embodiments can be completed by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When the program is executed, the steps of the above-mentioned method embodiments are executed; and the aforementioned storage medium includes: ROM, RAM, disk or optical disk and other media that can store program codes.

[0174] Finally, it should be noted that those skilled in the art will readily conceive of other embodiments of the present invention after considering the specification and practicing the invention disclosed herein. The present invention is intended to cover any variations, uses or adaptations of the present invention, which follow the general principles of the present invention and include common knowledge or customary technical means in the art not disclosed by the present invention, are not limited to the precise structure described above and shown in the drawings, and may be modified and changed in various ways without departing from the scope thereof. The scope of the present invention is limited only by the appended claims.

Claims

1. A method for generating a three-dimensional virtual character, characterized in that: include: Acquire generation conditions of a three-dimensional virtual character, wherein the generation conditions include text description information and / or image description information; Encoding the generation condition to obtain corresponding generation condition features; the generation condition features include a first face structure feature, a first face texture feature and a face occlusion feature; The generation condition feature is processed using a three-dimensional character generation model to obtain a second face texture feature and a second face structure feature; wherein the three-dimensional character generation model is trained based on a plurality of face image groups and image information of each face image group, the face image group includes images of the same face under different illuminations and different postures, and the image information includes facial attribute descriptions and identity identifiers; The second facial texture feature and the second facial structure feature are input into a three-dimensional rendering engine to obtain a target three-dimensional virtual character output by the three-dimensional rendering engine.

2. The method according to claim 1, characterized in that The three-dimensional character generation model is used to process the generation condition feature to obtain a second face texture feature and a second face structure feature, including: Performing texture reconstruction processing on the first facial texture feature and / or the facial occlusion feature to obtain the second facial texture feature corresponding to the generation condition; wherein the first facial texture feature is obtained by processing the texture text in the text description information through a text encoder; and the facial occlusion feature is obtained by expanding the image description information; The first facial structure feature is geometrically reconstructed to obtain a second facial structure feature corresponding to the generation condition; wherein the first facial structure feature is obtained by processing the geometric text in the text description through a text encoder; or, when the generation condition only includes image description information, the second facial structure feature is estimated using a pre-trained face reconstruction model.

3. The method according to claim 2, characterized in that The performing texture reconstruction processing on the first facial texture feature and / or the facial occlusion feature to obtain the second facial texture feature corresponding to the generation condition includes: Processing the first face texture feature and / or the face occlusion feature using a texture diffusion model to obtain a third face texture feature corresponding to the generation condition; wherein the texture diffusion model is trained based on a plurality of face image groups and image information of each face image group, the face image group includes images of the same face under different illuminations and different postures, and the image information includes facial attribute description and identity identification; The resolution of the third facial texture feature is adjusted using a pre-trained super-resolution network to obtain a texture that meets the target resolution as the second facial texture feature.

4. The method according to claim 3, characterized in that The using of the texture diffusion model to process the first facial texture feature and / or the facial occlusion feature to obtain a third facial texture feature corresponding to the generating condition includes: Encode the face occlusion feature using a pre-trained encoder to obtain an incomplete latent variable corresponding to the face occlusion feature; Adding the first facial texture feature to the texture diffusion model by means of cross attention as a first guiding condition of the texture diffusion model; Taking a given first Gaussian noise as a noise latent variable, concatenating it with the incomplete latent variable, and inputting it into the texture diffusion model, so that the texture diffusion model predicts and denoises the noise latent variable based on the first guiding condition and the incomplete latent variable, and outputs a texture latent variable corresponding to the first guiding condition; The texture latent variable is decoded using a pre-trained decoder to obtain a third face texture feature corresponding to the generation condition.

5. The method according to claim 2, characterized in that: The step of geometrically reconstructing the first facial structure feature to obtain the second facial structure feature corresponding to the generation condition includes: The first facial structure feature is added to a geometric diffusion model by cross-attention as a second guiding condition of the geometric diffusion model; wherein the geometric diffusion model is trained based on a plurality of facial image groups and image information of each facial image group, wherein the facial image group includes images of the same face under different illuminations and different postures, and the image information includes facial attribute description and identity identification; Inputting a given second Gaussian noise into the geometric diffusion model, so that the geometric diffusion model performs predictive denoising on the second Gaussian noise based on the second guiding condition to obtain an identity coefficient corresponding to the first facial structure feature; The identity coefficient is subjected to basis linear weighting processing to obtain a second facial structure feature corresponding to the generation condition.

6. The method according to any one of claims 1 to 5, characterized in that: The method for constructing the plurality of face image groups and the image information of each group of face images includes: Acquire multiple face images; For each of the face images, delighting and relighting are performed to obtain a delighted multi-view face image and a relighted face image; Based on the de-illuminated multi-view face image and the re-illuminated face image, images under different illuminations and different postures, and an albedo texture UV map corresponding to the face image are obtained, and a correspondence is established between the images under different illuminations and different postures, and the albedo texture UV map and the face image to obtain the face image group; Selecting a frontal view face image from the de-illuminated multi-view face images, performing identity coefficient estimation and facial attribute description processing on the frontal view face image, and obtaining a facial attribute description corresponding to the face image and an identity identifier corresponding to the face image as image information of the face image; For each of the facial images, a correspondence is established between the facial image group and the image information to obtain the multiple facial image groups and the image information of each facial image group.

7. A three-dimensional virtual character generation device, characterized in that: include: An acquisition module, used to acquire generation conditions of a three-dimensional virtual character, wherein the generation conditions include text description information and / or image description information; An encoding module, used for encoding the generation condition to obtain corresponding generation condition features; the generation condition features include a first face structure feature, a first face texture feature and a face occlusion feature; a condition processing module, used to process the generation condition feature using a three-dimensional character generation model to obtain a second face texture feature and a second face structure feature; wherein the three-dimensional character generation model is trained based on a plurality of face image groups and image information of each face image group, wherein the face image group includes images of the same face under different illuminations and different postures, and the image information includes facial attribute description and identity identification; A generation module is used to input the second facial texture feature and the second facial structure feature into a three-dimensional rendering engine to obtain a target three-dimensional virtual character output by the three-dimensional rendering engine.

8. An electronic device, characterized in that: include: Memory, processor; The memory stores computer-executable instructions; The processor executes the computer-executable instructions stored in the memory, so that the processor performs the method according to any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer-executable instructions, which are used to implement the method according to any one of claims 1 to 6 when executed by a processor.

10. A computer program product, comprising a computer program, wherein when the computer program is executed by a processor, the method according to any one of claims 1 to 6 is implemented.

Images