A multi-angle multi-style face portrait generation method with explicit view angle control

Abstract

The application discloses a multi-angle multi-style face portrait generation method with controllable visual angle, and comprises the following steps: S1, constructing a face image dataset; S2, randomly initializing a 256-dimensional vector subject to a standard normal distribution, and mapping the 256-dimensional vector into a frequency, phase and modulation vector through a multilayer perception machine; S3, constructing a face portrait generation model; S4, obtaining a multi-channel feature map; S5, obtaining a high-resolution semantic map and a portrait; and S6, obtaining a multi-angle explicit controllable multi-style face portrait. The method mainly relates to inputting 3D information as an input of a generative adversarial network, so that the generative adversarial network is 3D perceptible, and then the visual angle of a generated object is explicitly controlled, and then through 2D level style migration, the generated face portrait is stylized into a sketch, a pen sketch, a line drawing and an oil painting, and the explicit control of the visual angle is achieved.

Description

Technical Field

[0001] This invention relates to computer image processing, specifically to a method for generating multi-angle, multi-style facial portraits with controllable viewpoint display. It mainly involves using 3D information as input and output of a generative adversarial network (GAN), making the GAN 3D-aware, thereby explicitly controlling the viewpoint of the generated object. Then, through 2D style transfer, the generated facial portrait is stylized using techniques such as sketching, pen drawing, line drawing, and oil painting, achieving explicit control over the viewpoint. Background Technology

[0002] The explicit control of viewpoint is mainly applied in scene reconstruction, which is to reconstruct the entire scene in 3D using visual information from several different viewpoints, thereby obtaining visual information from various angles. The introduction of neural radiation fields has introduced a more efficient way for scene reconstruction, which is to sample points on the light rays in space and obtain the pixel values ​​of the final viewpoint by sampling these points along the light rays. Each position in space is represented by density and color values, and the number of points sampled directly affects the quality of reconstruction.

[0003] Traditional generative adversarial networks (GANs) reflect a function mapping process from a standard normal distribution to a pixel value distribution. By introducing 3D information into GANs, making them perceptible to 3D information, the function mapping process changes to a process from a standard normal distribution to a voxel value distribution in space. This transformation allows for explicit control over the viewing angle of the generated object or human face portrait. Changing the starting position of the viewing angle allows for explicit control over the viewing angle of the final 2D image.

[0004] The task of generating multi-angle, multi-style facial portraits mainly faces two problems:

[0005] In stylized portraiture, such as sketches, pen drawings, and line drawings, artists employ techniques like perspective and composition. For instance, in a sketch portrait, features like the cheeks and forehead are not elaborated upon; their color in the finished sketch is simply the color of the paper. When the perspective of the portrait changes, the color of the same area will change. Figure 1 The sketches in the image illustrate this problem. In the same face, the dot at the bridge of the nose is white from a frontal view, but when the view changes to a profile, the dot at the same location becomes a line, and the color turns black. This demonstrates that because sketches cannot consistently represent the same face in three-dimensional space, this style of portraiture cannot be directly applied to 3D perceptual generative adversarial networks.

[0006] The amount of multi-style portrait data is limited, such as sketches and pen drawings. There are very few portraits of these styles. With too little data, the model's generated results will be limited to a small amount of training data, and the diversity of the generated results cannot be guaranteed. At the same time, the style portraits in the limited dataset have little variation in perspective, and most of them are frontal views. This will cause the model to be unable to control the perspective of the generated results. That is, the training dataset needs to contain style portraits from different perspectives. Otherwise, the model will not be able to generate style portraits with diverse perspectives when it encounters unfamiliar perspectives. Summary of the Invention

[0007] The problem this invention aims to solve is to propose a viewpoint-controllable multi-angle, multi-style face portrait generation method. It expands the number of existing portraits of different styles. Then, considering that 2D style images such as sketches and pen drawings cannot be directly applied to 3D perception generative adversarial networks, a portrait generation network and a semantic graph generation network are designed. A two-step training process is adopted to complete the training of the viewpoint-controllable multi-angle, multi-style face portrait generation network.

[0008] This invention proposes a method for generating multi-angle, multi-style facial portraits with explicit viewpoint control, comprising the following steps:

[0009] S1. Construct a face image dataset;

[0010] S2. Randomly initialize a 256-dimensional vector that follows a standard normal distribution and map it to frequency, phase, and modulation vectors through a multilayer perceptron;

[0011] S3. Construct a face portrait generation model, which includes a feature generation network, a semantic graph generation network, and a portrait generation network;

[0012] S4. Sampling points of the face image in the cubic space, each point contains three values ​​of three-dimensional coordinates x, y, and z. The sampled points in the space are put into the feature map generation network used to control the display viewpoint, and the frequency and phase obtained in S2 are used in the operation of the activation function to obtain a multi-channel feature map.

[0013] S5. The feature map obtained in S4 and the modulation vector obtained in S2 are put into the semantic map generation network and the portrait generation network respectively, and the high-resolution semantic map and portrait obtained by the generation network are constrained by the portrait discrimination network and the semantic map discrimination network.

[0014] S6. Lock the feature generation network in S4, and train the portrait generation network and semantic graph generation network under the styles of sketch, line drawing, pen drawing and oil painting. Then, constrained by the portrait discrimination network and semantic graph discrimination network, obtain multi-angle explicit and controllable multi-style face portrait images.

[0015] Preferably, in step S2, the process of obtaining the frequency, phase, and modulation vector is as follows:

[0016] 1-1 A 256-dimensional vector that follows a standard normal distribution is mapped to a modulation vector using a 3-layer Linear network + LeakyReLU(0.2) structure. The dimension of the modulation vector is 256.

[0017] 1-2 The obtained modulation vector is input into a 1-layer Linear network to obtain 9 sets of frequencies and phases.

[0018] Preferably, in step S4, the sampling method for spatial points is as follows: the -z axis direction is set as the observation direction, the sampling range of the z axis is set to [0.88, 1.12], and the spatial resolution is set to 64*64. That is, along the z axis, 4096 light rays are sampled, and 16 points are set on each light ray. Each point contains three values: x, y, and z, for a total of 16*64*64=65536 sampling points.

[0019] Preferably, the feature generation network comprises four network layers: the first layer is an 8-layer Linear network with a sin() structure, where sin() is the activation function; the second layer is a 1-layer Linear network; the third layer is a 1-layer Linear network with a sin() structure, where sin() is the activation function; and the fourth layer is a 1-layer Linear network with a sigmoid structure, where sigmoid is the activation function.

[0020] Preferably, in step S4, the method for mapping the sampled spatial points to a feature map is as follows:

[0021] The obtained spatial points are processed through an 8-layer Linear network with a sin() structure to obtain intermediate outputs. In the sin activation function, the output of the Linear network is multiplied by the frequency obtained in S2, and then added to the phase in S2. The intermediate output is processed through a 1-layer Linear network to obtain the density values ​​of different points in space. The intermediate output is then processed through a 1-layer Linear network with a sin() structure to obtain multi-layer feature maps. The multi-layer feature maps are then processed through a 1-layer Linear network with a sigmoid structure to obtain multi-layer color maps. Finally, the multi-layer color maps, multi-layer feature maps, and density values ​​are integrated to obtain a 256*64*64 feature map, a 3*64*64 portrait map, and a 1*64*64 depth map.

[0022] Preferably, in step S5, the process of obtaining the high-resolution semantic map and portrait is as follows:

[0023] 5-1 To obtain a high-resolution portrait, the modulation vector obtained in S2 is passed through a 1-layer Linear network to obtain the mean and variance. The feature map obtained in S4 is standardized in the channel dimension, then multiplied by the variance and added to the mean to complete the modulation of the feature map. After one convolution and activation operation, the feature map is upsampled twice by 2 times to obtain a 256*256 resolution face portrait.

[0024] 5-2 To obtain a high-resolution semantic map, the modulation vector obtained in S2 is passed through a 1-layer Linear network to obtain the mean and variance. The feature map obtained in S4 is standardized in the channel dimension, then multiplied by the variance and added to the mean to complete the modulation of the feature map. After a convolution and activation operation, the feature map is upsampled twice by 2x, and the number of output channels is set to 19 to obtain a 19*256*256 semantic map.

[0025] As a preferred embodiment, the process of generating multi-angle, multi-style facial portraits in step S6 is as follows:

[0026] 6-1 Lock the feature generation network. When training the overall network, set the optimization parameters of the optimizer, exclude the parameters in the feature generation network, and optimize only the semantic graph generation network and the portrait generation network. Set separate optimizers for the portrait discrimination network and the semantic graph discrimination network.

[0027] 6-2 The portrait generation network is trained using sketch portraits. The sketch portraits are obtained by transferring the face portrait generation model trained in S5 through 2D sketching algorithm. In the portrait generation network, the output of the semantic graph generation network is used as input to assist in the generation of sketch portraits. The generated sketch portraits are semantically consistent with the semantic graph. At the same time, when the semantic graph is edited, the generated sketch portraits can correspond to the edited semantic graph and undergo corresponding changes. The portrait generation network is constrained by the discriminant network.

[0028] 6-3 uses line art to train the portrait generation network. The line art portrait generation process is consistent with the sketch portrait generation process in 4-2 and is directly controlled by the semantic map.

[0029] 6-4 uses pen drawings to train the portrait generation network, and the training process and method are the same as those in 6-2 and 6-3;

[0030] 6-5 Oil paintings are used to train the portrait generation network. The oil painting dataset is collected manually. When training the portrait generation network with oil paintings, the semantic graph generation network is not used as input, and it is only constrained by the portrait discrimination network.

[0031] The generation angles of the portrait generation network and the semantic map generation network in 6-6 are controlled by the feature map generation network locked in 6-1. The angle of the feature map directly controls the angle of the subsequent portrait and semantic map. The explicit control of the angle is determined by the spatial feature map established in S4.

[0032] The advantages of this invention compared to existing 3D perceptual generative adversarial networks are:

[0033] (1) The expressive ability of traditional 3D perception generative adversarial networks can only be used to generate three-dimensional structures that can be realistically expressed in space. However, the portrait generation network proposed in this invention can generate not only human face portraits, but also 2D style human face portraits in sketch, line drawing, and pen drawing. It can also explicitly control the perspective of the generated portrait. By changing the position of the camera, multiple styles of human face portraits can be obtained from the corresponding perspective. Furthermore, this invention can ensure as much as possible that the identity structure of the portrait will not change when the perspective changes.

[0034] (2) This invention solves the problem of insufficient multi-style portrait data. It adopts a 2D level multi-style transfer method to convert the original face portrait dataset into sketch, pen drawing and line drawing styles, which provides data support for the training of the network structure proposed in this invention. Attached Figure Description

[0035] Figure 1 This is a schematic diagram illustrating the problem of applying stylistic portraiture to the 3D perceptual image generation task.

[0036] Figure 2 This is a schematic diagram illustrating the implementation method of an embodiment of the present invention.

[0037] Figure 3 It shows the detailed structure of Blocks in the portrait generation network and semantic graph generation network, as well as the structural diagram of semantic graph-assisted portrait graph generation;

[0038] Figure 4 These are renderings of the model proposed in this invention in different styles;

[0039] Figure 5 The text in the figure shows the editing effect of this method. Detailed Implementation

[0040] To enable those skilled in the art to better understand the technical solutions of this disclosure, the following detailed description of this disclosure is provided in conjunction with the accompanying drawings and specific embodiments.

[0041] This embodiment presents a method for generating multi-view, explicitly controllable, multi-angle, and multi-style facial portraits, such as... Figure 2 As shown, the steps are as follows:

[0042] S1. Map a 256-dimensional vector that follows a standard normal distribution to frequency, phase, and modulation vectors;

[0043] S2. Sampling of points in the cubic space, each point containing three values ​​(x, y, z). The sampled points in the space are put into the feature map generation network for explicit control of the viewpoint, and the frequency and phase obtained in S1 are used in the operation of the activation function to obtain a multi-channel feature map.

[0044] S3. The feature map obtained in S2 and the modulation vector obtained in S1 are respectively put into the semantic map generation network and the portrait generation network, and the semantic map and portrait obtained by the generation network are constrained by the portrait discrimination network and the semantic map discrimination network to generate high-resolution semantic maps and portraits.

[0045] S4. Lock the feature generation network in S2, and train the portrait generation network and semantic graph generation network using images in the styles of sketch, line drawing, pen drawing and oil painting. The generation network is constrained by the discriminator network to obtain multi-angle explicit and controllable multi-style face portrait images.

[0046] Specifically, the modulation module is used to map a vector that follows a standard normal distribution into a modulation vector, frequency, and phase;

[0047] Feature generation network is used to map points in space and modulated frequencies and phases into feature maps in space;

[0048] Semantic graph generation network and semantic graph discrimination network. The semantic graph generation network is constrained by the semantic graph discrimination network to generate semantic graphs from various perspectives.

[0049] The portrait generation network and the portrait discrimination network are constrained by the portrait discrimination network. The portrait generation network generates portrait images from various angles and is modulated by the semantic graph when generating multi-style portrait images.

[0050] Example:

[0051] A method for generating multi-angle, multi-style facial portraits with explicit viewpoint control includes the following steps:

[0052] S1. Map a 256-dimensional vector that follows a standard normal distribution to a modulation vector, frequency, and phase.

[0053] In this embodiment, the original 256-dimensional vector following a standard normal distribution in step S1 can be directly generated by a standard method. The distribution of the 256 values ​​follows a standard normal distribution. After obtaining the original vector, a 3-layer Linear + LeakyReLU(0.2) module is used to operate on the original vector. Each layer of the Linear module has both an input and output dimension of 256. After three operations, a modulation vector is obtained, also with a dimension of 256. A copy of the modulation vector is saved here, specifically located at... Figure 2 As shown in the diagram, the modulation vector is then passed through a Linear layer to obtain multiple frequencies and phases. Each frequency and phase has a dimension of 256, including 9 groups. The input dimension of the last Linear layer is 256, and the output dimension is 9*256*2. Through the operation of the last layer, 9 groups of frequencies and phases are obtained, which are used by the subsequent feature generation network.

[0054] S2. Sampling is performed on the midpoint of the space, a feature generation network is used, and the 9 sets of frequencies and phases obtained in S1 are used for the activation operation to obtain a multi-channel feature map.

[0055] In this embodiment, the randomly sampled points in space contain three values ​​(x, y, z). The first layer of the feature generation network has an input dimension of 3 and an output dimension of 256. The input and output dimensions of all intermediate layers are 256. In each layer, the frequency and phase obtained in S1 are multiplied and added. After eight layers of such operations, an intermediate output is obtained. The intermediate output is then processed by a Linear layer to map the output dimension to 1, obtaining the density value in space. After the intermediate output passes through the last Linear+sin structure, a multi-layer feature map is obtained. This is then processed by another Linear layer to map the output dimension to 3, and the sigmoid activation function is selected for activation, resulting in a multi-layer color map. Finally, the multi-layer feature map, multi-layer color map, and density value are integrated to obtain the final color map, feature map, and depth map. The number of channels and size of the feature map are 256*64*64, the number of channels and size of the color map are 3*64*64, and the number of channels of the depth map is 1, with a size of 64*64. The form of the feature map is as follows: Figure 2 The data is shown in the image; multiple channels are selected for heatmap display.

[0056] Figure 2 FiLM is the name of the linear layer and the sine activation function structure, and there are a total of 9 such structures.

[0057] S3. The feature map obtained in S2 and the modulation vector obtained in S1 are put into the semantic graph generation network and the portrait generation network respectively, and the semantic graph discrimination network and the portrait discrimination network are used for constraints to generate high-resolution portraits and semantic maps.

[0058] In this embodiment, the method for convolution and modulation of the feature image involves first passing the modulation vector obtained in S1 through a Linear layer to obtain the mean and variance, then performing multiplication and addition operations to obtain the modulated features, and finally performing a convolution and activation operation. This single operation corresponds to... Figure 2 Block in the middle corresponds to Figure 3 The detailed structure of the Block, after two passes, yields a 256*256 resolution portrait image and semantic map. In training steps S1, S2, and S3, the semantic map generation network runs parallel to the portrait generation network, and the semantic map does not assist in portrait generation. Only in step S4 is the semantic map used as input to the portrait generation network to assist in the generation of multi-style face portraits. This step corresponds to... Figure 2 The semantic graph in the image is input into the truncated arrow part of the portrait generation network.

[0059] S4. The feature generation network from S2 is locked in place. Images in sketch, line drawing, pen drawing, and oil painting styles are used to train the portrait generation network and the semantic graph generation network. The generation network is constrained by the discriminative network, resulting in multi-angle, explicitly controllable, multi-style facial portraits. The generated images can be seen... Figure 4 .

[0060] In this embodiment, Figure 2 The feature map generation network and modulation module are locked. Specifically, the optimizer's optimization parameters are set only for the semantic map generation network and the portrait generation network, optimizing only these two networks. Meanwhile, a separate optimizer is set for the discriminator network. Then, the semantic map auxiliary module in the portrait generation network is activated. Figure 3 The semantic graph modulation part, aided by the semantic graph, enables the portrait generation network to possess rich expressive capabilities while also having a certain degree of editing functionality. It can adjust the semantic range generated from different angles based on changes in the semantic graph. The editing effect is shown in [the image / description]. Figure 5 .

[0061] In this embodiment, when selecting portrait data of different styles, sketch, pen drawing, line drawing, and oil painting datasets, which are obtained by style transfer from the original portrait data, are used for training respectively. Each style can obtain a network model. During training, except for the oil painting dataset which is not generated without the assistance of semantic graphs, the other three styles can use semantic graphs for image editing. During the training process, the optimizer used is the Adam optimizer, which is the same as the optimizer trained in S3. The learning rate decreases accordingly as the resolution increases. In the two upsampling stages and three phases, the learning rates are set to 6e-5, 5e-5, and 3e-5 respectively. When performing style transfer in the S4 stage, the third stage, i.e., training at a resolution of 256*256, is performed directly with a learning rate of 3e-5. Finally, multi-angle images of four styles are obtained.

[0062] The parts of this invention not described in detail are well-known to those skilled in the art.

Claims

1. A method for generating multi-angle, multi-style facial portraits with explicitly controllable perspective, characterized in that, The steps include the following: S1. Construct a face image dataset; S2. Randomly initialize a 256-dimensional vector that follows a standard normal distribution and map it to frequency, phase, and modulation vectors through a multilayer perceptron; S3. Construct a face portrait generation model, which includes a feature generation network, a semantic graph generation network, and a portrait generation network; S4. Sampling is performed on the face image in a cubic space. Each point contains three values ​​of x, y, and z three-dimensional coordinates. The sampled points in the space are put into a feature generation network for display control of the viewing angle. The frequency and phase obtained in S2 are used as activation function operations to obtain a multi-channel feature map. The feature generation network includes four network layers. The first network layer is an 8-layer Linear network + sin() structure, where sin() is the activation function; the second network layer is a 1-layer Linear network; the third network layer is a 1-layer Linear network + sin() structure, where sin() is the activation function; the fourth network layer is a 1-layer Linear network + sigmoid structure, where sigmoid is the activation function. S5. The feature map obtained in S4 and the modulation vector obtained in S2 are put into the semantic graph generation network and the portrait generation network respectively, and the portrait and high-resolution semantic map obtained by the generation network are constrained by the portrait discrimination network and the semantic graph discrimination network. To obtain a high-resolution portrait: The modulation vector obtained in S2 is passed through a 1-layer Linear network to obtain the mean and variance. The feature map obtained in S4 is standardized in the channel dimension, then multiplied by the variance and added to the mean to complete the modulation of the feature map. After one convolution and activation operation, the feature map is upsampled twice by 2 times to obtain the face portrait. To obtain a high-resolution semantic map: The modulation vector obtained in S2 is passed through a 1-layer Linear network to obtain the mean and variance. The feature map obtained in S4 is standardized in the channel dimension, then multiplied by the variance and added to the mean to complete the modulation of the feature map. After a convolution and activation operation, the feature map is upsampled twice by 2 times. The number of output channels is set to obtain the semantic map. S6. Lock the feature generation network in S4, and train the portrait generation network and semantic graph generation network under the styles of sketch, line drawing, pen drawing and oil painting. Then, constrained by the portrait discrimination network and semantic graph discrimination network, obtain multi-angle explicit and controllable multi-style face portrait images.

2. The method for generating multi-angle, multi-style facial portraits with explicit viewpoint controllability according to claim 1, characterized in that, In step S2, the process of obtaining the frequency, phase, and modulation vector is as follows: 1-1 A 256-dimensional vector that follows a standard normal distribution is mapped to a modulation vector using a 3-layer Linear network + LeakyReLU(0.2) structure. The modulation vector has a dimension of 256. 1-2 The obtained modulation vector is input into a 1-layer Linear network to obtain 9 sets of frequencies and phases.

3. The method for generating multi-angle, multi-style facial portraits with explicit viewpoint controllability according to claim 1, characterized in that, In step S4, the sampling method for spatial points is as follows: the -z axis direction is set as the observation direction, the sampling range of the z axis is set along the z axis, and n rays are sampled, where n is the product of the image resolution. 16 points are set on each ray, and each point contains three values: x, y, and z, for a total of 16*n sampling points.

4. The method for generating multi-angle, multi-style facial portraits with explicit viewpoint controllability according to claim 3, characterized in that, In step S4, the method for mapping the sampling points in the sampled space to a feature map is as follows: The sampling points in the obtained space are processed through an 8-layer Linear network + sin() structure to obtain an intermediate output. In the sin activation function, the output of the Linear network is multiplied by the frequency obtained in S2, and then added to the phase in S2. Passing the intermediate output through a single-layer Linear network yields density values ​​at different points in space. Passing the intermediate output through a single-layer Linear network with a sin() structure yields multi-layer feature maps. Passing the multi-layer feature maps through a single-layer Linear network with a sigmoid structure yields multi-layer color maps. Finally, integrating the multi-layer color maps, multi-layer feature maps, and density values ​​yields feature maps, portrait maps, and depth maps.

5. The method for generating multi-angle, multi-style facial portraits with explicit viewpoint controllability according to claim 1, characterized in that, In step S6, the process of generating multi-angle, multi-style facial portraits is as follows: 6-1 Lock the feature generation network. When training the overall network, set the optimization parameters of the optimizer, exclude the parameters in the feature generation network, and optimize only the semantic graph generation network and the portrait generation network. Set separate optimizers for the portrait discrimination network and the semantic graph discrimination network. 6-2 The portrait generation network is trained using sketch portraits. The sketch portraits are obtained by transferring the face data of the face portrait generation model trained in S5 through a 2D sketching algorithm. In the portrait generation network, the output of the semantic graph generation network is used as input to assist in the generation of the sketch portrait. The generated sketch portrait is semantically consistent with the semantic graph. At the same time, when the semantic graph is edited, the generated sketch portrait can correspond to the edited semantic graph and undergo corresponding changes. The portrait generation network is constrained by the discriminant network. 6-3 uses line art to train the portrait generation network. The line art portrait generation process is consistent with the sketch portrait generation process in 4-2 and is directly controlled by the semantic map. 6-4 uses pen drawings to train the portrait generation network, and the training process and method are the same as those in 6-2 and 6-3; 6-5 Oil paintings are used to train the portrait generation network. The oil painting dataset is collected manually. When training the portrait generation network with oil paintings, the semantic graph generation network is not used as input, and it is only constrained by the portrait discrimination network. The generation angles of the portrait generation network and the semantic map generation network in 6-6 are controlled by the feature map generation network locked in 6-1. The angle of the feature map directly controls the angle of the subsequent portrait and semantic map. The explicit control of the angle is determined by the spatial feature map established in S4.

Images