A depth information guided multi-style face sketch generation method

Abstract

The present invention discloses a method for generating multi-style face sketches guided by depth information, comprising the following steps: step (1) data preprocessing, aligning, scaling, and cropping face photos and face sketches, and then generating corresponding depth images from the face photos in the data set according to the 3DDFA method, ultimately obtaining a one-to-one corresponding face photo-face sketch-face depth map triple; step (2) constructing a generative adversarial network model, respectively constructing two network models, a generator and a discriminator; step (3) training the generative adversarial network model, defining a loss function, and training the generative adversarial network model; and step (4) generating and evaluating the quality of face sketches. The present invention proposes a method for controlling the generation of face sketches using face depth information and style features, thereby achieving high-quality, multi-style generation of face sketches, and can be successfully applied to generation tasks of non-face sketches such as natural images.

Description

Technical Field

[0001] The present invention relates to image processing, and specifically mentions a new method for generating multi-style face sketches guided by depth information (Face2 Sketch Generation Guided by Facial Depth Map), which mainly involves using facial depth information and style features to construct a multi-style face sketch generation model that is more consistent with real painting effects. Background Art

[0002] Many image generation problems involve converting an input image into a corresponding output image, such as converting between grayscale images, gradient images, and color images. In particular, in the field of facial image generation, applications include generating pencil and pen drawings of faces. Facial sketch generation is also essentially an image generation problem, with the ultimate goal of achieving high-quality conversion from a facial photograph to a corresponding sketch image.

[0003] Currently popular facial image generation algorithms, such as pix2pix, Genre, and cycleGan, are effective at generating facial sketches. However, the resulting sketches are of low quality, and there is still significant room for improvement in capturing local facial details. For example, these methods may not be able to handle certain variations in appearance, such as pose, lighting, expression, and skin color. Furthermore, these methods are only suitable for generating sketches in a single style and cannot address the generation of sketches in different styles. Summary of the Invention

[0004] The purpose of the present invention is to address the problem of face sketch generation quality and to provide a multi-style face sketch generation method guided by depth information.

[0005] The present invention solves the technical problem and provides a depth information-guided multi-style face sketch generation method, comprising the following steps:

[0006] Step (1) Construct a dataset and preprocess the image data in the dataset

[0007] Step (2) Build a generative adversarial network model and generate a face sketch

[0008] 2-1 Generator builds and generates feature vectors

[0009] The structure of the generator G follows the classic U-Net structure, connecting the output of the i-th encoding layer and the output of the corresponding i-th decoding layer. Each encoding layer includes a convolution layer, a dynamic activation layer, and an instance normalization layer; each decoding layer includes an upsampling convolution layer, a dynamic adaptive activation layer, and a dynamic adaptive modulation layer.

[0010] 2-2 through depth and style adaptive normalization processing;

[0011] 2-3 Adaptive Dynamic Activation

[0012] 2-4 Deformable Contour Generation

[0013] 2-5 Discriminator construction and output

[0014] The discriminator consists of 6 layers, namely convolutional layer, instance normalization layer, and activation layer. The input of the discriminator is a vector of the face depth map and the face sketch concatenated by channel. For the face depth map and the real face sketch, the discriminator output is True; for the face depth map and the generated face sketch, the discriminator output is False.

[0015] Step (3) Generate adversarial network model training

[0016] Define the loss function and train the generative adversarial network model;

[0017] Step (4) Generation and quality evaluation of face sketches

[0018] The face photo of the sketch to be generated is input into the trained generative adversarial network model to obtain the corresponding face sketch image, and the quality evaluation is completed according to the indicators.

[0019] Preferably, the preprocessing method is: perform face alignment, scaling and cropping on the face photos and face sketches in the data set, and then generate corresponding depth images for the face photos in the data set according to the 3DDFA method, and finally obtain a one-to-one corresponding face photo-face sketch-face depth map triplet.

[0020] Preferably, after preprocessing, the size of the face photo is 250*250 and the number of channels is 3; the size of the face sketch is 250*250 and the number of channels is 1; the size of the face depth map is 250*250 and the number of channels is 1.

[0021] Preferably, in the generator, the first two decoding layers use deformable convolution.

[0022] Preferably, the feature vector generated by the generator includes depth information D of the face, facial style information S and facial appearance features A.

[0023] Preferably, the depth and style adaptive normalization processing method is as follows:

[0024] By scaling and biasing the features, the depth information D of the face, the style information S of the face and the appearance feature A of the face extracted by the encoder are used. Assume that F∈R C×H×WRepresents the input features of the current IDN module, where H, W, and C represent the height, width, and number of channels of the feature, respectively. Then the feature values ​​after modulation at the c, h, and w positions are:

[0025]

[0026] Among them, f c,h,w and They are the eigenvalues ​​before and after the input of the IDN module, μ c and σ c are f in the cth channel respectively c,h,w The mean and standard deviation of :

[0027]

[0028]

[0029] Among them, γ c,h,w (D, S, A) and β c,h,w (D, S, A) are the scaling and bias parameters learned through shallow neural network, and the specific calculation process is as follows:

[0030] γ=Conv(metaACON(Conv(Cat(D,S,A)))),

[0031] β=Conv(metaACON(Conv(Cat(D,S,A)))).

[0032] Preferably, the adaptive dynamic activation method is as follows:

[0033] Inspired by the decision of whether to draw a stroke based on 3D geometry, 2D appearance, and style type, we improved the existing metaACON dynamic activation function so that the parameter θ is dynamically learned based on the depth information D of the face, the facial style information S, and the facial appearance features A extracted by the encoder. The original dynamic activation function metaACON is as follows:

[0034] y=(p1-p2)·σ(θ(p1-p2)x)+p2x

[0035] Where y represents the output after activation, θ = σ(x), σ is a Sigmoid function, p1 and p2 are learnable parameters; and the adaptive dynamic activation function IDA is as follows:

[0036] θ=σ(θ(Cat(D,S,A))),

[0037] y=(p1-p2)·σ(θ(p1-p2)x)+p2x

[0038] Here, σ represents a shallow neural network.

[0039] Preferably, the deformable contour generating method is as follows:

[0040] There is geometric deformation between the completed facial sketch and the real photo. Feature alignment is performed on a coarse scale. Deformable convolution is used instead of standard transposed convolution in the first two layers of the decoding layer. The input features are first amplified by two times through bilinear interpolation and then input into a deformable convolution layer. This module significantly improves the clarity of the generated contour.

[0041] Preferably, the loss function defined in step (3) is as follows:

[0042]

[0043]

[0044]

[0045]

[0046]

[0047]

[0048]

[0049] L all =L adv +λ1×L pix +λ2×L geo +λ3×L tex

[0050] Among them, Y s Represents a real face sketch image, represents the face sketch image generated by the model, D represents the real face depth image, Represents the face depth image reconstructed by the model, g i,j Represents Y s The gradient in the x and y directions at the coordinate (i, j), g i,j Represents The gradient in the x and y directions at the coordinate (i, j), D(D,Y s ) indicates that the input is D and Y s The output of the discriminator is Indicates that the input is and The output of the discriminator; M and N represent the width and height of the sketch;

[0051] L adv represents the generation adversarial loss function, L pix represents the pixel-level difference between the generated sketch and the real sketch, represented by and Y s The L1 distance between them is obtained; L geo Represents the pixel-level difference between the reconstructed depth map and the real depth map, represented by The L1 distance between L and D is obtained; tex represents texture loss, requiring the generated sketch to have a texture similar to the real sketch; L all Represents the total loss function, which is composed of L adv and L pix 、L geo 、L tex The three loss functions are summed by multiplying weights λ1, λ2, and λ3 respectively.

[0052] Preferably, the generation and quality evaluation method in step (4) is:

[0053] The facial photos in the dataset are input into the trained generative adversarial network model in turn to obtain the generated face sketch image. The generated image is compared with the real sketch to calculate the FID index and complete the quality evaluation of the generated sketch.

[0054] The beneficial effects of the present invention are as follows:

[0055] Compared with existing facial sketch generation methods, the depth-information-guided multi-style facial sketch generation method described in this invention significantly improves the quality of facial sketch generation, particularly in terms of producing more realistic results in local brushstroke details. Furthermore, this method can generate facial sketches of different styles. The following table compares various metrics calculated for facial sketch generation results using this method and several existing methods in the field of image generation. Lower FID and LPIPS values ​​indicate higher quality, while higher SCOOT and FSIM values ​​indicate higher quality. BRIEF DESCRIPTION OF THE DRAWINGS

[0056] Figure 1 This is the architecture diagram of the face sketch generation adversarial network model.

[0057] Figure 2 This is the architecture diagram of the encoder layer of the face sketch generator.

[0058] Specific implementation details

[0059] The present invention will be further described below with reference to the accompanying drawings.

[0060] like Figure 1 、 Figure 2 As shown in FIG, a depth information guided multi-style face sketch generation method specifically includes the following steps:

[0061] Step (1) Data preprocessing

[0062] Perform face alignment, scaling, and cropping on the face photos and face sketches, and then generate corresponding depth images based on the 3DDFA method for the face photos in the dataset, ultimately obtaining a one-to-one correspondence of face photo-face sketch-face depth map triplet;

[0063] Specifically, 1-1 selects the FS2K dataset, and according to the official dataset division method, selects 1058 face photos and corresponding sketch images as the training set, and the remaining 1046 face images and corresponding sketch images as the test set. Among them, Style 1 has 357 face images and corresponding sketch images as the training set and 619 face images and corresponding sketch images as the test set; Style 2 has 351 face images and corresponding sketch images as the training set and 381 face images and corresponding sketch images as the test set; Style 3 has 350 face images and corresponding sketch images as the training set and 46 face images and corresponding sketch images as the test set;

[0064] 1-2 Preprocess the images in the dataset (including face photos and sketch images). First, align the key points of the images, and then interpolate and scale them to 250*250 size. The number of channels of face photos is 3, and the number of channels of sketch images is 1.

[0065] 1-3 generates a corresponding face depth image based on the 3DDFA depth generation method for the face photos in the dataset preprocessed in 1-2. The image has the same size of 250*250 as the face photo and the number of channels is 1. The size of the pixel value in the image represents the depth information of the position.

[0066] Step (2) Construction of Generative Adversarial Network Model

[0067] Specifically, the generative adversarial network model includes a generator and a discriminator. The generator adopts an encoder-decoder structure, and the discriminator is a convolutional neural network.

[0068] 2-1 Generator Construction

[0069] The structure of the generator G follows the classic U-Net structure. We connect the output of the i-th encoding layer to the output of the corresponding i-th decoding layer. Each encoder layer consists of a convolutional layer (Conv), a dynamic activation layer (metaACON), and an instance normalization layer (IN), namely (Conv-metaACON-IN). Each decoder layer consists of an upsampling convolution (TrConv / DOG), a dynamic adaptive activation layer (IDA), a dynamic adaptive modulation layer (IDN), and a module, namely (TrConv / DOG-IDA-IDN). The first two decoding layers use deformable convolution (DOG), and the remaining layers use ordinary convolution (TrConv).

[0070] 2-2 Depth and Style Adaptive Normalization Module

[0071] First, to better modulate neurons, we implemented a depth and style adaptive normalization (IDN) modulation module. We modulate features by scaling and biasing. Unlike the existing SPADE adaptive normalization modulation, we use the depth information D of the face, the face style information S, and the face appearance feature A extracted by the encoder. Assume F∈R C×H×W Represents the input features of the current IDN module, where H, W, and C represent the height, width, and number of channels of the feature, respectively. Then the feature value after modulation at the (c, h, w) position is:

[0072]

[0073] Among them, f c,h,w and They are the eigenvalues ​​before and after the input of the IDN module, μ c and σ c are f in the cth channel respectively c,h,w The mean and standard deviation of :

[0074]

[0075]

[0076] In particular, the γ c,h,w (D, S, A) and β c,h,w (D, S, A) are the scaling and bias parameters learned through shallow neural network, and the specific calculation process is as follows:

[0077] γ=Conv(metaACON(Conv(Cat(D,S,A)))),

[0078] β=Conv(metaACON(Conv(Cat(D,S,A))))

[0079] 2-3 Adaptive Dynamic Activation Module

[0080] Inspired by how artists decide whether to draw a stroke based on 3D geometry, 2D appearance, and style, we improved the existing metaACON dynamic activation function and implemented an adaptive dynamic activation module (Informative and Dynamic Activation, IDA) to dynamically learn the parameters θ based on the face's depth information D, facial style information S, and facial appearance features A extracted by the encoder. The original dynamic activation function metaACON is as follows:

[0081] y=(p1-p2)·σ(θ(p1-p2)x)+p2x

[0082] Where y represents the output after activation, θ = σ(x), σ is a Sigmoid function, p1 and p2 are learnable parameters; and the adaptive dynamic activation function IDA is as follows:

[0083] θ=σ(θ(Cat(D,S,A))),

[0084] y=(p1-p2)·σ(θ(p1-p2)x)+p2x

[0085] Among them, σ represents a shallow neural network (Conv-metaACON-Conv).

[0086] 2-4 Deformable Contour Generation Module

[0087] We observed geometric distortions between artist-created facial sketches and real-world photographs. To mimic the artist's abstract drawing style, we designed a Deformable Outline Generation (DOG) module to perform feature alignment at a coarse scale. We used deformable convolutions (DCNs) instead of standard transposed convolutions in the first two layers of the decoder. Specifically, we first upscaled the input features by a factor of two using bilinear interpolation before feeding them into a deformable convolutional layer. This module significantly improved the clarity of the generated outlines.

[0088] 2-5 Discriminator Construction

[0089] The input of the discriminator is a face depth map and a face sketch (real sketch Y s / Generate Sketch ) is a channel-wise concatenated vector. The discriminator consists of six layers: a convolutional layer (Conv), an instance normalization layer (IN), and an activation layer (LeakyReLU), or (Conv-IN-LeakyReLU). The discriminator is required to output True for a face depth map and a real face sketch, and False for a face depth map and a generated face sketch.

[0090] Step (3) Generate adversarial network model training

[0091] Define the loss function and train the generative adversarial network model;

[0092] 3-1 When training a generative adversarial network, the training of the generative adversarial network is completed when the loss function reaches the minimum; the loss function is as follows:

[0093]

[0094]

[0095]

[0096]

[0097]

[0098]

[0099]

[0100] L all =L adv +λ1×L pix +λ2×L geo +λ3×L tex

[0101] Among them, Y s Represents a real face sketch image, represents the face sketch image generated by the model, D represents the real face depth image, Represents the face depth image reconstructed by the model, g i,j Represents Y s The gradient in the x and y directions at the coordinate (i, j), g i,j Represents The gradient in the x and y directions at the coordinate (i, j), D(D,Y s ) indicates that the input is D and Y s The output of the discriminator is Indicates that the input is and The output of the discriminator when ; M and N represent the width and height of the sketch.

[0102] L adv Represents the generation of adversarial loss function, here we use Cross Entropy loss; L pix represents the pixel-level difference between the generated sketch and the real sketch, represented by and Y s The L1 distance between them is obtained; L geo Represents the pixel-level difference between the reconstructed depth map and the real depth map, represented by The L1 distance between L and D is obtained; tex represents texture loss, requiring the generated sketch to have a texture similar to the real sketch; L all Represents the total loss function, which is composed of L adv and L pix 、L geo 、L tex The three loss functions are summed by weights λ1, λ2, and λ3. During the training process, we train the generator G and the discriminator D, L all To minimize.

[0103] FID LPIPS SCOOT FSIM Pix2Pix 18.34 0.304 0.493 0.541 Pix2PixHD 32.03 0.468 0.374 0.531 CycleGAN 26.49 0.505 0.348 0.501 MDAL 50.18 0.492 0.355 0.530 SCA-GAN 39.63 0.305 0.600 0.782 FSGAN 34.88 0.483 0.405 0.610 GENRE 20.67 0.302 0.483 0.534 Ours 15.06 0.263 0.575 0.551

[0104] Step (4) Generation and quality evaluation of face sketches

[0105] Input the face photo of the person whose sketch is to be generated into the trained generative adversarial network model to obtain the corresponding face sketch image, and complete the quality evaluation according to the indicators;

[0106] Specifically, the facial photos in the test set are input into the trained adversarial generative network in turn to obtain the generated facial sketch images. The generated images are then compared with the real sketches to calculate indicators such as FID, LPIPS, SCOOT, and FSIM to complete the quality evaluation of the generated sketches.

Claims

1. A depth information guided multi-style face sketch generation method, characterized by: The steps include: Step (1) Construct a dataset and preprocess the image data in the dataset Step (2) Build a generative adversarial network model and generate a face sketch 2-1 Generator builds and generates feature vectors The structure of the generator G follows the classic U-Net structure, connecting the output of the i-th encoding layer and the output of the corresponding i-th decoding layer. Each encoding layer includes a convolution layer, a dynamic activation layer, and an instance normalization layer; each decoding layer includes an upsampling convolution layer, a dynamic adaptive activation layer, and a dynamic adaptive modulation layer. 2-2 Through depth and style adaptive normalization processing, the method is as follows: By scaling and biasing the features, the depth information D of the face, the style information S of the face and the appearance feature A of the face extracted by the encoder are used. Assume that F∈R C×H×W Represents the input features of the current IDN module, where H, W, and C represent the height, width, and number of channels of the feature, respectively. Then the feature values ​​after modulation at the c, h, and w positions are: Among them, f c,h,w and They are the eigenvalues ​​before and after the input of the IDN module, μ c and σ c are f in the cth channel respectively c,h,w The mean and standard deviation of : Among them, γ c,h,w (D, S, A) and β c,h,w (D, S, A) are the scaling and bias parameters learned through shallow neural network, and the specific calculation process is as follows: γ=Conv(metaACON(Conv(Cat(D,S,A)))), β=Conv(metaACON(Conv(Cat(D,S,A)))); 2-3 adaptive dynamic activation; 2-4 Deformable contour generation; 2-5 discriminator is constructed and output, The discriminator consists of 6 layers, namely convolutional layer, instance normalization layer, and activation layer. The input of the discriminator is a vector of the face depth map and the face sketch concatenated by channel. For the face depth map and the real face sketch, the discriminator output is True; for the face depth map and the generated face sketch, the discriminator output is False. Step (3) Generate training of adversarial network model, Define the loss function and train the generative adversarial network model; Step (4) Generation and quality evaluation of face sketches, The face photo of the sketch to be generated is input into the trained generative adversarial network model to obtain the corresponding face sketch image, and the quality evaluation is completed according to the indicators.

2. The depth information guided multi-style face sketch generation method according to claim 1 is characterized in that: The preprocessing method is as follows: face photos and face sketches in the dataset are aligning, scaling and cropping the faces, and then generating corresponding depth images for the face photos in the dataset according to the 3DDFA method, and finally obtaining a one-to-one corresponding face photo-face sketch-face depth map triplet.

3. The depth information guided multi-style face sketch generation method according to claim 2, characterized in that: After preprocessing, the size of the face photo is 250*250 and the number of channels is 3; the size of the face sketch is 250*250 and the number of channels is 1; the size of the face depth map is 250*250 and the number of channels is 1.

4. The depth information guided multi-style face sketch generation method according to claim 1, characterized in that: In the generator, the first two decoding layers use deformable convolution.

5. The depth information guided multi-style face sketch generation method according to claim 1, characterized in that: The feature vector generated by the generator includes depth information D of the face, facial style information S and facial appearance features A.

6. The depth information guided multi-style face sketch generation method according to claim 5, characterized in that: The adaptive dynamic activation method is as follows: Inspired by deciding whether to draw a stroke based on 3D geometry, 2D appearance, and style type, we improved the existing metaACON dynamic activation function so that the parameter θ is dynamically learned based on the depth information D of the face, the facial style information S, and the facial appearance features A extracted by the encoder. The adaptive dynamic activation function IDA is as follows: y=(p1-p2)·σ(θ(p1-p2)x)+p2x θ=σ(θ(Cat(D,S,A))) Among them, y represents the output after activation, p1 and p2 are learnable parameters; σ represents a shallow neural network.

7. The depth information guided multi-style face sketch generation method according to claim 6, characterized in that: The deformable contour generation method is as follows: There is geometric deformation between the completed facial sketch and the real photo. Feature alignment is performed on a coarse scale. Deformable convolution is used instead of standard transposed convolution in the first two layers of the decoding layer. The input features are first amplified by two times through bilinear interpolation and then input into a deformable convolution layer. This module significantly improves the clarity of the generated contour.

8. The depth information guided multi-style face sketch generation method according to claim 7, characterized in that: The loss function defined in step (3) is as follows: L all =L adv +λ1×L pix +λ2×L geo +λ3×L tex Among them, Y s Represents a real face sketch image, represents the face sketch image generated by the model, D represents the real face depth image, Represents the face depth image reconstructed by the model, g i,j Represents Y s The gradient in the x and y directions at the coordinate (i, j), g i,j Represents The gradient in the x and y directions at the coordinate (i, j), D(D,Y s ) indicates that the input is D and Y s The output of the discriminator is Indicates that the input is and The output of the discriminator; M and N represent the width and height of the sketch; L adv represents the generation adversarial loss function, L pix represents the pixel-level difference between the generated sketch and the real sketch, represented by and Y s The L1 distance between them is obtained; L geo Represents the pixel-level difference between the reconstructed depth map and the real depth map, represented by The L1 distance between L and D is obtained; tex represents texture loss, requiring the generated sketch to have a texture similar to the real sketch; L all Represents the total loss function, which is composed of L adv and L pix , L geo , L tex The three loss functions are summed by multiplying weights λ1, λ2, and λ3 respectively.

9. The depth information guided multi-style face sketch generation method according to claim 8, characterized in that: Generation and quality evaluation method in step (4): The facial photos in the dataset are input into the trained generative adversarial network model in turn to obtain the generated face sketch image. The generated image is compared with the real sketch to calculate the FID index and complete the quality evaluation of the generated sketch.

Images