A face photo-sketch synthesis method based on two-path conditional normalization

Abstract

The application discloses a face photo-sketch image synthesis method based on double-path condition normalization, comprising the following steps: obtaining a face photo-sketch image pair to be synthesized; inputting the face photo-sketch image pair to be synthesized into a trained face photo-sketch image synthesis network to obtain a synthesis result; wherein the face photo-sketch image synthesis network comprises an encoder, a generator and a semantic segmentation module, the generator comprises a double-path normalization module, a gated attention feature fusion module and a decoder; and the trained face photo-sketch image synthesis network is obtained through training of a face photo-sketch image pair training set. The application improves the quality of face photo-sketch image synthesis effect.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of artificial intelligence and image processing, and particularly relates to a face photo-sketch portrait synthesis method based on double-path conditional normalization. BACKGROUND

[0002] With the rapid development of the field of computer vision, face photo-sketch portrait synthesis technology has gradually become a research hotspot. Face photo-sketch portrait synthesis is the process of synthesizing a face photo into a face sketch portrait or synthesizing a face sketch portrait into a face photo. After such a conversion process, we can convert the photo information and portrait information in different domains into the same domain. And since hand-drawn face sketch portraits require professional artists and a lot of time, computer image processing algorithms that automatically synthesize face sketch portraits from face photos have high application value.

[0003] Early generation methods were mostly sample-based methods, and were one-way generation from photos to sketches. These methods use the idea of ​​nearest neighbor block matching, but the generated results often have a lot of blur and artifacts. Zhang et al. proposed an end-to-end fully convolutional network (FCN) in the paper "L.Zhang, L.Lin, X.Wu, S.Ding, and L.Zhang, "End-to-end photo-sketch generation via fully convolutional representation learning," in Proceedings of the 5thACM on International Conference on Multimedia Retrieval, 2015, pp.627–634." to directly learn the mapping between face photos and sketches; Isola et al. proposed an end-to-end fully convolutional network (FCN) in the paper "P.Isola, J.-Y.Zhu, T.Zhou, and AAEfros, "Image-to-image translation with conditional adversarial networks," in Proceedings of the IEEE conference on computer vision and patternrecognition,2017,pp.1125–1134.” proposed a general method called pix2pix for image-to-image conversion on paired datasets; Zhu et al. proposed the CycleGAN network framework in the document “J.-Y.Zhu,T.Park,P.Isola,and AAEfros,“Unpairedimage-to-image translation using cycle-consistent adversarial networks,”inProceedings of the IEEE international conference on computer vision,2017,pp.2223–2232.”, which can form a universal mapping from domain A to domain B and learn how to convert between the two domains, rather than being restricted to a specific image conversion. It can be trained using unpaired datasets and has strong adaptability; Chen et al. proposed the CycleGAN network framework in the document “C.Chen,W.Liu,X.Tan,and K.-YKWong, “Semi-supervised learning for face sketch synthesis in the wild,” in Asian Conference on Computer Vision. Springer, 2018, pp. 216–231. proposed a semi-supervised learning method to expand photo-sketch pairs by constructing pseudo-portrait features of additional training photos; Yu et al. proposed in the paper “J. Yu, X. Xu, F. Gao, S. Shi, M. Wang, D. Tao, and Q. Huang, “Toward realistic face photo–sketch synthesis via composition-aided gans,” IEEE transactions on cybernetics, vol. 51, no. 9, pp. 4350–4362, 2020.” to use facial component information as supplementary input and introduce part loss to focus training on specific parts; Nie et al. proposed in the paper “L. Nie, L. Liu, Z. Wu, and W. Kang, “Unconstrained face sketch synthesis via perception-adaptive network and a new A face sketch synthesis based on perceptual adaptive network under constrained and unconstrained conditions is proposed in "Neurocomputing, vol.494, pp.192–202, 2022."

[0004] However, the above existing methods do not encode sufficient texture and spatial information during training, and the visual quality of the generated images has some defects, such as blur, artifacts, missing details, and lack of sketch brushstrokes in the generated portraits. Summary of the Invention

[0005] In order to solve the above problems existing in the prior art, the present invention provides a face photo-sketch synthesis method based on two-way conditional normalization. The technical problem to be solved by the present invention is achieved through the following technical solutions:

[0006] An embodiment of the present invention provides a face photo-sketch synthesis method based on two-way conditional normalization, comprising:

[0007] Obtain a face photo-sketch pair to be synthesized;

[0008] Inputting the face photo-sketch pair to be synthesized into a trained face photo-sketch synthesis network to obtain a synthesis result;

[0009] The face photo-sketch synthesis network includes an encoder, a generator, and a semantic segmentation module. The generator includes a two-way normalization module, a gated attention feature fusion module, and a decoder. The trained face photo-sketch synthesis network is obtained by training a training set of face photo-sketch pairs. The corresponding training process includes:

[0010] The encoder encodes the face photo-sketch pair training set and outputs deep features; the semantic segmentation module extracts semantic labels of the sketches in the training set; the two-way normalization module strengthens the spatial information branch and the texture information branch according to the semantic labels and the deep features; the gated attention feature fusion module fuses the output results of the spatial information branch and the texture information branch; the decoder decodes the fusion result and outputs the synthesis result corresponding to the face photo-sketch pair training set; the loss function of the face photo-sketch synthesis network is constructed according to the face photo-sketch pair training set and its corresponding synthesis result; the parameters of the face photo-sketch synthesis network are updated according to the loss function, and training is continued until the iteration stop condition is met to obtain the trained face photo-sketch synthesis network.

[0011] In one embodiment of the present invention, the encoder comprises a plurality of convolutional layers connected in sequence;

[0012] In the encoder, the outputs corresponding to the last three convolutional layers of the encoder are used as deep features.

[0013] In one embodiment of the present invention, the dual-path normalization module includes a SPADE Resblock module and an AdaIN Resblock module, wherein:

[0014] The SPADE Resblock module enhances spatial information based on the semantic labels of the sketches in the training set and the depth features of the face photos in the training set;

[0015] The AdaIN Resblock module enhances texture information based on the depth features of sketches and face photos in the training set.

[0016] In one embodiment of the present invention, the AdaIN Resblock module includes a plurality of sequentially connected residual AdaIN modules; wherein each of the residual AdaIN modules includes a sequentially connected basic AdaIN module and a residual module.

[0017] In one embodiment of the present invention, the output of the SPADE Resblock module is expressed as:

[0018]

[0019] Among them, p represents a face photo, A i represents the output of the SPADEResblock module corresponding to the depth feature of the i-th layer (i=1,2,3), represents the depth feature corresponding to the face photo p in the i-th (i=1,2,3) layer depth feature, c represents the channel, (y,x) represents At a position on channel c, express The value at point (y,x) on channel c, M s represents a semantic label, Respectively On the c-th channel, from the semantic label M s The scaling and offset learned in Respectively Mean and variance on the c-th channel.

[0020] In one embodiment of the present invention, the output of the AdaIN Resblock module is expressed as:

[0021]

[0022] Among them, s represents a sketch, B i represents the output of the AdaINResblock module corresponding to the i-th (i=1,2,3) layer depth feature, represents the depth feature corresponding to the sketch image s in the i-th (i=1,2,3) layer depth feature, express On channel c The corresponding value at the same point (y,x), Respectively represent F s i Mean and variance on the c-th channel.

[0023] In one embodiment of the present invention, the gated attention feature fusion module includes two gate modules and a channel attention module; wherein,

[0024] The outputs of the SPADE Resblock module and the AdaIN Resblock module are respectively input into two gating modules;

[0025] The outputs of the SPADE Resblock module and the AdaIN Resblock module are superimposed and input into the channel attention module;

[0026] The outputs of the two gating modules are fused with the output of the channel attention module.

[0027] In one embodiment of the present invention, the outputs of the two gating modules and the output of the channel attention module are fused and expressed as:

[0028]

[0029] Among them, C i represents the output of the gated attention feature fusion module corresponding to the depth feature of the i-th layer (i=1, 2, 3), CA(·) represents the channel attention function, Indicates A i The corresponding gating function, Indicates B i The corresponding gating function.

[0030] In one embodiment of the present invention, the decoder is a decoder based on an AFF module;

[0031] In the decoder, after upsampling the output of the gated attention feature fusion module to obtain features of the same resolution, the AFF module is used to perform feature fusion and decode the output.

[0032] In one embodiment of the present invention, the loss function of the constructed face photo-sketch synthesis network is expressed as:

[0033] L full =λ1L adversarial +λ2L cycle +λ3L perceptual ;

[0034] Among them, L full represents the loss function of the face photo-sketch synthesis network; L adversarial represents the generation adversarial loss, L cycle represents the cycle consistency loss, L perceptual represents the perceptual loss, and λ1, λ2, and λ3 represent the balance parameters.

[0035] Beneficial effects of the present invention:

[0036] The present invention proposes a face photo-sketch portrait synthesis method based on dual-path conditional normalization, and proposes a face photo-sketch portrait synthesis network including a dual-path normalization module and a gated attention feature fusion module. The branches of the dual-path normalization module fully encode texture and spatial information, strengthen the learning of spatial information and texture information, and can retain more realistic edge and detail features. The gated attention feature fusion module fully filters and fuses the useful information of the two branches, avoiding information redundancy to a certain extent. The two together improve the quality of the face photo-sketch portrait synthesis effect. According to user voting surveys, the users have a better subjective feeling about the synthesized image of the embodiment of the present invention, and have a better user experience.

[0037] The present invention will be further described in detail below with reference to the accompanying drawings and embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] Figure 1 1 is a flow chart of a method for synthesizing a face photo and a sketch image based on two-way conditional normalization provided by an embodiment of the present invention;

[0039] Figure 2 This is a schematic diagram of the structure of a face photo-sketch synthesis network provided by an embodiment of the present invention;

[0040] Figure 3 1 is a schematic structural diagram of a dual-path conditional normalization module provided by an embodiment of the present invention;

[0041] Figure 4 is a schematic diagram of the structure of a gated attention feature fusion module provided by an embodiment of the present invention;

[0042] Figure 5 Schematic diagram of the training process of a face photo-sketch image synthesis network provided by an embodiment of the present invention;

[0043] Figure 6 1 is a schematic diagram comparing the sketch image synthesis results of the method provided by the embodiment of the present invention and six existing methods on the CUFS dataset, CUFSF dataset and WildSketch dataset respectively;

[0044] Figure 7 1 is a schematic diagram comparing the face photo synthesis results of the method provided by the embodiment of the present invention and three existing methods on the CUFS dataset and the CUFSF dataset respectively;

[0045] Figure 8 1. This is a schematic diagram comparing the satisfaction voting results of user surveys conducted using the method provided by the embodiment of the present invention and three existing methods using the face photo synthesis results of the CUFS dataset and the CUFSF dataset respectively;

[0046] Figure 9 It is a structural diagram of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0047] The present invention will be further described in detail below with reference to specific examples, but the embodiments of the present invention are not limited thereto.

[0048] To improve the quality of face photo-sketch synthesis, see Figure 1 The embodiment of the present invention provides a face photo-sketch portrait synthesis method based on two-way conditional normalization, which specifically includes the following steps:

[0049] S10, obtaining a face photo-sketch pair to be synthesized.

[0050] Specifically, in the face photo-sketch portrait pair to be synthesized according to the embodiment of the present invention, if the face photo is synthesized through the sketch portrait, a reference face photo will be randomly selected, and the reference face photo and the sketch portrait will form the face photo-sketch portrait pair to be synthesized; if the sketch portrait is synthesized through the face photo, a reference sketch portrait will be randomly selected, and the face photo and the reference sketch portrait will form the face photo-sketch portrait pair to be synthesized. It can be seen that the face photo-sketch portrait synthesis method according to the embodiment of the present invention can realize both the synthesis of sketch portraits and the synthesis of face photos. Among them, the randomly selected reference face photo or reference sketch portrait can provide a large amount of texture and spatial prior information, thereby improving the quality of the final face photo-sketch portrait synthesis effect.

[0051] S20, inputting the face photo-sketch pair to be synthesized into a trained face photo-sketch synthesis network to obtain a synthesis result.

[0052] Specifically, see Figure 2 The face photo-sketch synthesis network proposed in the embodiment of the present invention includes an encoder, a generator and a semantic segmentation module. The generator includes a two-way normalization module, a gated attention feature fusion module and a decoder. The synthesis process and the training process use the same network structure, and the same network structure is used to realize the synthesis of sketches and the synthesis of face photos during synthesis or training, except that the input and output are different. The internal processing process of the network is similar. Therefore, it is not clear in the following description whether the sample is a sample to be synthesized or a training sample. For each face photo-sketch pair in the training set, or face photo-sketch pair to be synthesized, for example Figure 2 The specific design of each part is as follows:

[0053] The encoder of the embodiment of the present invention includes several sequentially connected convolutional layers, and extracts multi-scale depth features through a deep convolutional network; specifically, in the encoder, the outputs corresponding to the last three convolutional layers of the encoder are used as depth features. Specifically:

[0054] For the input sketch image s, it passes through an encoder composed of a series of convolutional layers and performs forward propagation. The deep feature representations of different resolutions are output at the last three levels of the encoder as Among them, i=1,2,3 represents the three levels of output features, c i 、h i 、w i They represent the number of channels, height, and width of the feature map of the i-th layer respectively.

[0055] For the input face photo p, it passes through the same encoder as the sketch image s (the encoder shares parameters) and performs forward propagation. The deep feature representation of different resolutions output at the last three levels of the encoder is Among them, i=1,2,3 represents the three levels of output features, c i 、h i 、w i They represent the number of channels, height, and width of the feature map of the i-th layer respectively.

[0056] Here, if the face photo is to be synthesized through the sketch image s, then Figure 2 The face photo p in the image is a randomly selected reference face photo p ref , if it is a synthesis process and the sketch portrait s is synthesized through the face photo p, then Figure 2 Sketch s is a randomly selected reference sketch s ref Similarly, if the training process is to train the face-sketch synthesis network by using the sketch image s, then Figure 2 The face photo p in the training set is a random reference face photo p ref If the training process is to train the face photo-sketch synthesis network using the face photo p, then Figure 2 The sketch portrait s is a random reference face photo s in the training set ref .

[0057] Furthermore, the embodiment of the present invention extracts the depth features and Perform two-way normalization. The two-way normalization operation consists of two conditional normalization branches, which explicitly decompose the overall mapping into two independent mappings, respectively strengthening the learning of spatial information and texture information. Figure 3In view of this design idea, the embodiment of the present invention designs a dual-path normalization module, which includes two branches consisting of a SPADE Resblock module and an AdaIN Resblock module, wherein:

[0058] The first spatial information branch is implemented by the SPADE Resblock module, which enhances spatial information based on the semantic labels of the sketches and the depth features of the facial photos in the training set. The second texture information branch is implemented by the AdaIN Resblock module, which enhances texture information based on the depth features of the sketches and facial photos in the training set.

[0059] For the first spatial information branch, the SPADE Resblock module of the embodiment of the present invention can directly use the existing SPADE Resblock module. The SPADE Resblock module is described in detail in “T. Park, M.-Y. Liu, T.-C. Wang, and J.-Y. Zhu, “Semantic image synthesis with spatially-adaptive normalization,” in Proceedings of the IEEE / CVF conference on computer vision and pattern recognition, 2019, pp. 2337–2346.” and will not be described in detail here. The specific network structure of the SPADE Resblock module is as follows: Figure 3 The present invention uses the semantic segmentation model BiSeNet trained on the CelebA-HQ database to extract the semantic label M of the input sketch image s. s , then and M s Input into the SPADE Resblock module, the channel level is labeled with semantic labels M s The mean and variance learned in Modulation is performed to enhance the learning of spatial information. The output of the final SPADE Resblock module is expressed as:

[0060]

[0061] Among them, p represents a face photo, A i represents the output of the SPADE Resblock module corresponding to the depth feature of the i-th layer (i=1,2,3), represents the depth feature corresponding to the face photo p in the i-th (i=1,2,3) layer depth feature, c represents the channel, (y,x) represents At a position on channel c, express The value at point (y,x) on channel c, M s represents a semantic label, Respectively On the c-th channel, from the semantic label M s The scaling and offset learned in Respectively The mean and variance on the cth channel.

[0062] The mean and variance on the cth channel are expressed as:

[0063]

[0064]

[0065] Among them, H i 、W i Respectively The corresponding total width and total height.

[0066] However, due to the semantic label M s Only semantic information is available. Using this branch alone will inevitably lose fine texture information. Therefore, the embodiment of the present invention also designs another branch, the AdaIN Resblock module, to enhance texture information.

[0067] For the second texture information branch, the embodiment of the present invention is improved on the basis of the existing AdaIN module. The existing AdaIN module is described in detail in "X. Huang and S. Belongie, "Arbitrary style transfer in real-time with adaptive instance normalization," in Proceedings of the IEEE international conference on computer vision, 2017, pp. 1501–1510.", and will not be repeated here. Since the AdaIN module is a real-time style transfer module, it is introduced to adjust the mean and variance to The style information is passed to Enhance the representation ability of texture.

[0068] However, since the modulation gap between the face photo and the sketch portrait is large, and the adaptive ability of the non-parametric operation is insufficient, directly using the existing AdaIN module is not suitable for completing this task. Therefore, the embodiment of the present application improves the learning ability by introducing a residual block, please see again Figure 3 The AdaIN Resblock module of the embodiment of the present application includes a plurality of residual AdaIN modules connected in sequence; wherein each residual AdaIN module includes a basic AdaIN module and a residual module connected in sequence. For example, the AdaIN ResBlock module of the embodiment of the present application is composed of 9 residual AdaIN modules connected in sequence, each residual AdaIN module is composed of an existing AdaIN module and a residual block, and the residual block added after the existing AdaIN module can improve the receptive field and enhance the adaptive ability of the branch. The output of the final AdaIN Resblock module is represented as:

[0069]

[0070] Wherein s represents the sketch portrait, B i represents the output of the AdaIN Resblock module corresponding to the i(i=1, 2, 3) layer deep feature, represents the deep feature corresponding to the sketch portrait s in the i(i=1, 2, 3) layer deep feature, represents the value of the same point (y, x) corresponding to respectively represent the mean and variance on the c-th channel. Wherein,

[0071] The mean and variance on the c-th channel are calculated similarly respectively represented as:

[0072]

[0073]

[0074] Wherein, the total width and total height corresponding to formula (5) and formula (6) are calculated respectively. Here, the total width and total height corresponding to respectively take values of H i , W i , and the total width and total height corresponding to

[0075] ​Furthermore, the present invention proposes a gated channel attention fusion module, which integrates the gating mechanism and the channel attention mechanism and fuses the two information branches, allowing the network to selectively amplify useful feature channels and suppress useless feature channels based on global information. Figure 4 , the gated attention feature fusion module of the embodiment of the present invention includes two gate modules and a channel attention module; wherein,

[0076] The outputs of the SPADE Resblock module and the AdaIN Resblock module are input into two gating modules respectively; the outputs of the SPADE Resblock module and the AdaIN Resblock module are superimposed and input into the channel attention module; the outputs of the two gating modules are fused with the output of the channel attention module. Specifically, the fusion of the outputs of the two gating modules and the output of the channel attention module is expressed as:

[0077]

[0078] Among them, A i and B i They represent the output of the SPADE Resblock module and the output of the AdaIN Resblock module corresponding to the i-th (i=1,2,3) layer depth feature, respectively. “·” represents the element-by-element multiplication. Figure 3 Chinese Indicates that "+" represents element-by-element addition. Figure 3 Chinese Indicates that C i represents the output of the gated attention feature fusion module corresponding to the deep features of the i-th layer (i=1,2,3), CA(·) represents the channel attention function, Indicates A i The corresponding gating function, Indicates B i The corresponding gating function. Figure 3 Middle gate control module input A i Corresponding output Gate module input B i Corresponding output Figure 3 α in CA(A i +B i ).

[0079] Here, the gating function Respectively expressed as:

[0080]

[0081]

[0082] Among them, Conv represents a 1×1 convolution operation, and σ represents a sigmoid function.

[0083] The channel attention function CA(·) is expressed as:

[0084] CA(A i +B i )=σ(Conv2(δ(Conv1(A i +B i )))) (10)

[0085] Among them, Conv1 and Conv2 both represent 1×1 convolution operations, and δ represents the rectified linear unit ReLU.

[0086] The gating function of the embodiment of the present invention can determine the importance of each eigenvector in the feature map and effectively control the information flow. The channel attention can explicitly model the interdependence between channels and globally filter the information flow, thereby allowing the network to selectively amplify useful feature channels and suppress useless feature channels based on global information.

[0087] Further, see Figure 2 The decoder of the embodiment of the present invention adopts a decoder based on the AFF module. In the decoder, the output of the gated attention feature fusion module is upsampled to obtain features of the same resolution, and then the AFF module is used to perform feature fusion and decode the output. Specifically:

[0088] The outputs of the three-layer gated attention feature fusion module are integrated using a decoder based on attention feature fusion (AFF) to generate the final synthesis result. The AFF module is an existing attention feature fusion module, which is introduced in detail in "Y.Dai, F.Gieseke, S.Oehmcke, Y.Wu, and K.Barnard, "Attentional feature fusion," in Proceedings of the IEEE / CVF Winter Conference on Applications of Computer Vision, 2021, pp.3560–3569." and will not be repeated here. In this embodiment of the present invention, after upsampling the three feature maps obtained by the gated attention feature fusion module to the same resolution, the AFF module is used to perform feature fusion in pairs, and the final synthesis result is generated by decoding, which is a face photo. or sketch portrait

[0089] Furthermore, the trained face photo-sketch portrait synthesis network used in the synthesis process of the embodiment of the present invention is obtained by training a face photo-sketch portrait pair training set; wherein, M images are selected from the face photo dataset to form a face photo training set, and M face sketch portraits corresponding to the M face photos are selected from the face sketch portrait dataset to form M pairs of face photo-sketch portrait pairs, as the face photo-sketch portrait pair training set.

[0090] Please attend Figure 5 ,The detailed training process includes the following steps:

[0091] S201, the encoder encodes the face photo-sketch pair training set and outputs deep features;

[0092] S202, a semantic segmentation module extracts semantic labels of sketch images in the training set;

[0093] S203, the dual-path normalization module performs enhancement according to the spatial information branch and the texture information branch based on the semantic label and the depth feature;

[0094] S204, the gated attention feature fusion module fuses the output results of the spatial information branch and the texture information branch;

[0095] S205, the decoder decodes the fusion result and outputs the synthesis result corresponding to the face photo-sketch pair training set;

[0096] For the implementation of S201 to S205, please refer to the description of the specific design of each part above.

[0097] S206. Construct a loss function of a face photo-sketch synthesis network based on a training set of face photo-sketch pairs and their corresponding synthesis results.

[0098] Specifically, the loss function of the face photo-sketch synthesis network constructed in this embodiment of the present invention is expressed as:

[0099] L full =λ1L adversarial +λ2L cycle +λ3L perceptual (11)

[0100] Among them, L full represents the loss function of the face photo-sketch synthesis network; L adversarial represents the generation adversarial loss, L cycle represents the cycle consistency loss, L perceptualRepresents the perceptual loss, λ1, λ2, and λ3 represent balancing parameters to balance different loss values. In the embodiment of the present invention, λ1=1, λ2=0.5, and λ3=0.5 are set. It can be seen that the loss function of the embodiment of the present invention consists of three parts:

[0101] Generative Adversarial Loss: Generative adversarial loss aims to guide the generator to achieve more realistic generation results. When calculating the generative adversarial loss, a discriminator is added after the generator. The specific discriminator structure refers to the discriminator in CycleGAN in the paper "J.-Y.Zhu,T.Park,P.Isola,and AAEfros,“Unpaired image-to-image translation using cycle-consistentadversarial networks,”in Proceedings of the IEEE international conference on computer vision,2017,pp.2223–2232.” and is trained. The final generative adversarial loss for the generator and discriminator is expressed as:

[0102] L adversarial =E p～Pdata(p) [(D(p)) 2 ]+E s～Pdata(s) [(1-D(G(s,p ref ,M(s)))) 2 ] (12)

[0103] Among them, E(·) represents the expected value of the distribution function, E p～Pdata(p) (·) represents the expectation corresponding to the face photo p in the face photo dataset, E s～Pdata(s) (·) represents the expectation corresponding to the sketch s in the sketch dataset, D(·) represents the discriminator, D(p) represents the discrimination result obtained by inputting the face photo p into the discriminator, M(·) represents the face semantic segmentation network BiSeNet, M(s) represents the semantic label result obtained by inputting the sketch s into the semantic segmentation network, G(·) represents the generator that generates face photos through the sketch s, G(s,p ref ,M(s)) represents the sketch image s, the reference sample photo p ref , the synthesis result obtained by inputting the semantic label M(s) into the generator, that is, Figure 2 Generate photos D(G(s,p ref ,M(s))) represents the generated photo Input the discrimination result obtained in the discriminator.

[0104] Cycle consistency loss: Given a sketch s and its face label representation M(s), after the cyclic transformations S→P and P→S in the S domain and P domain, the sketch s should be transformed back to the original domain. The corresponding cycle consistency loss is expressed as:

[0105] L cycle =E s～Pdata(s) [||F(G(s,p ref ,M(s)),s ref ,M(G(s,p ref ,M(s))))-s||1] (13)

[0106] Among them, s ref 、p ref Denote the reference sample images of S domain and P domain respectively, G(s,p ref ,M(s)) represents the photo generated by the generator M(G(s,p ref ,M(s)))) means that photos will be generated Input the semantic label result obtained from the semantic segmentation network, F(·) represents the generator that generates sketch portraits from face photos, F(G(s,p ref ,M(s)),s ref ,M(G(s,p ref ,M(s)))) represents G(s,p ref ,M(s)) generated photos Reference sample sketch portraits ref , semantic label M(G(s,p ref ,M(s))) inputs the sketch image synthesis result obtained in the generator, the network structure of F(·) and G(·) is the same, ||·||1 means to find the L1 norm.

[0107] Perceptual loss: Introducing perceptual loss can ensure that the generated photos are similar to real photos at the semantic feature level. Perceptual loss is designed based on the pre-trained VGG-19 model. The real photos and generated photos are input into the pre-trained VGG-19 model. The corresponding perceptual loss is expressed as:

[0108]

[0109] Among them, Φ j (·) represents the output feature map of the jth layer in the pre-trained VGG-19 model, C j 、H j and W j They represent the number of channels, height, and width of the output feature map of the jth layer respectively.

[0110] Furthermore, the network parameters of the face photo-sketch synthesis network are updated according to the loss function, and the training is continued until the iteration stop condition is met. Then:

[0111] S207: Output the current face photo-sketch image synthesis network as the trained face photo-sketch image synthesis network.

[0112] During the entire iterative calculation process, the gradient descent algorithm can be used to update the parameters until the face photo-sketch portrait synthesis network model reaches convergence, but it is not limited to the gradient descent algorithm.

[0113] It should be noted that, in the embodiments of the present invention, whether it is the synthesis of sketch portraits or the synthesis of face photos, the above-mentioned training process can be used to train the corresponding face photo-sketch portrait synthesis network model. During the training process, corresponding reference face photos or reference sketch portraits are selected for training, which can provide a large amount of texture and spatial prior information, so that the trained face photo-sketch portrait synthesis network model can better synthesize sketch portraits or face photos.

[0114] In order to verify the effectiveness of the face photo-sketch image synthesis method based on two-way conditional normalization provided by the embodiment of the present invention, the following experiments were conducted for verification.

[0115] 1. Simulation conditions

[0116] The present invention uses the PyTorch framework for simulation on an Intel(R) Xeon(R) Gold 6226R 2.90GHz CPU, an NVIDIA GeForce RTX 3090 GPU, and an Ubuntu 16.04 operating system. Training is performed on the CUFS dataset, the CUFSF dataset, and the WildSketch dataset.

[0117] The methods compared in the experiment are as follows:

[0118] One approach is FCN (Frequency Convolutional Network). (L. Zhang, L. Lin, X. Wu, S. Ding, and L. Zhang, “End-to-end photo-sketch generation via fully convolutional representation learning,” in Proceedings of the 5th ACM on International Conference on Multimedia Retrieval, 2015, pp. 627–634). This approach proposes an end-to-end fully convolutional network to directly learn the mapping from face photos to portraits. However, this network is too shallow to extract deep semantic information.

[0119] The second is pix2pix, the reference is "P.Isola, J.-Y.Zhu, T.Zhou, and AAEfros, "Image-to-image translation with conditional adversarial networks," in Proceedings of the IEEE conference on computer vision and pattern recognition, 2017, pp.1125–1134." This method uses conditional GAN ​​(cGAN) as a unified solution for image-to-image conversion on paired datasets.

[0120] The third is CycleGAN, with the reference being "J.-Y.Zhu,T.Park,P.Isola,and AAEfros,“Unpaired image-to-image translation using cycle-consistent adversarialnetworks,”in Proceedings of the IEEE international conference on computervision,2017,pp.2223–2232.” It proposes a CycleGAN network framework that can form a universal mapping from domain A to domain B and learn how to convert between the two domains, rather than being restricted to a specific image conversion. It can be trained using unpaired datasets and has strong adaptability.

[0121] Four is a semi-supervised learning based sketch synthesis method in the wild, denoted as Wild in the experiment, and the reference is "C. Chen, W. Liu, X. Tan, and K.-Y. K. Wong, "Semi-supervised learning for face sketch synthesis in the wild," in Asian Conference on Computer Vision. Springer, 2018, pp. 216-231.", which expands the photo-sketch pairs by constructing pseudo portrait features of additional training photos.

[0122] Five is SCAGAN, and the reference is "J. Yu, X. Xu, F. Gao, S. Shi, M. Wang, D. Tao, and Q. Huang, "Toward realistic face photo-sketch synthesis via composition-aided gans," IEEE transactions on cybernetics, vol. 51, no. 9, pp. 4350-4362, 2020.", which proposes to take facial component information as supplementary input and introduces part loss to concentrate on the training of specific parts.

[0123] Six is PANet, and the reference is "L. Nie, L. Liu, Z. Wu, and W. Kang, "Unconstrained face sketch synthesis via perception-adaptive network and a new benchmark," Neurocomputing, vol. 494, pp. 192-202, 2022.", which proposes a face sketch synthesis method based on perception-adaptive network under constrained and unconstrained conditions.

[0124] 2. Simulation content

[0125] Part of the face photo-sketch pairs in the CUFS dataset, the CUFSF dataset, and the WildSketch dataset are selected as face photo-sketch pairs to be synthesized, and the sketch synthesis results of the present method and six existing methods are as shown in Table 1. Figure 6 Specifically, as shown in Table 1, Figure 6The first row is the sketch image synthesis result of the cuhk sub-dataset in the CUFS dataset, the second row is the sketch image synthesis result of the ar sub-dataset in the CUFS dataset, the third row is the sketch image synthesis result of the xm2vts sub-dataset in the CUFS dataset, the fourth row is the sketch image synthesis result of the CUFSF dataset, and the fifth row is the sketch image synthesis result of the WildSketch dataset; Figure 7 The first row is the face photo synthesis result of the cuhk sub-dataset in the CUFS dataset, the second row is the face photo synthesis result of the ar sub-dataset in the CUFS dataset, the third row is the face photo synthesis result of the xm2vts sub-dataset in the CUFS dataset, and the fourth row is the face photo synthesis result of the CUFSF dataset. Figure 6 and Figure 7 In the figure, the Ground Truth on the far right represents the synthetic image corresponding to the Test Photo on the far left in the training set. The closer the synthetic results of all methods are to the Ground Truth, the better the synthetic effect of the method. Figure 6 and Figure 7 , the second column on the right is the synthesis result of the method used in the present invention.

[0126] From above Figure 6 、 Figure 7 It can be seen that the synthesis result of the method adopted in the embodiment of the present invention well restores texture information and structural information, retains more realistic edge and detail features, and effectively overcomes the blur and artifact problems.

[0127] At the same time, the embodiment of the present invention selects three existing methods and the method used in the present invention ( Figure 8 A user survey was conducted on the face photo synthesis results of the CUFS dataset and the WildSketch dataset using DCNP (denoted as DCNP in the figure). A total of 14 questionnaires were set up (7 on the CUFS dataset and 7 on the WildSketch dataset), and users voted on the results they were most satisfied with. A total of 700 votes were collected from 50 users. The specific percentage of votes for each method is shown in Figure 8 As shown, it can be seen that the method of the present invention has a better subjective feeling, which verifies the advanced nature of the method of the present invention.

[0128] To sum up, the embodiment of the present invention proposes a face photo-sketch portrait synthesis method based on dual-path conditional normalization, and proposes a face photo-sketch portrait synthesis network including a dual-path normalization module and a gated attention feature fusion module. The branches of the dual-path normalization module fully encode texture and spatial information, strengthen the learning of spatial information and texture information, and can retain more realistic edge and detail features. The gated attention feature fusion module fully filters and fuses the useful information of the two branches, avoiding information redundancy to a certain extent. The two together improve the quality of the face photo-sketch portrait synthesis effect. According to user voting surveys, the users have a better subjective feeling about the synthesized image of the embodiment of the present invention, and have a better user experience.

[0129] At the same time, the embodiment of the present invention introduces additional reference sample images (reference face photos p ref Or refer to the sketches ref ) to provide a large amount of texture and spatial prior information, further improving the quality of face photo-sketch portrait synthesis.

[0130] See Figure 9 , an embodiment of the present invention provides an electronic device, including a processor 901, a communication interface 902, a memory 903 and a communication bus 904, wherein the processor 901, the communication interface 902, and the memory 903 communicate with each other via the communication bus 904;

[0131] Memory 903, used for storing computer programs;

[0132] The processor 901 is used to implement the steps of the above-mentioned face photo-sketch portrait synthesis method based on two-way conditional normalization when executing the program stored in the memory 903.

[0133] An embodiment of the present invention provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, the steps of the above-mentioned face photo-sketch portrait synthesis method based on two-way conditional normalization are implemented.

[0134] As for the electronic device / storage medium embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0135] In the description of the present invention, it should be understood that the terms "first" and "second" are used for descriptive purposes only and should not be understood to indicate or imply relative importance or implicitly specify the number of the technical features indicated. Therefore, a feature specified as "first" or "second" may explicitly or implicitly include one or more of the features. In the description of the present invention, "plurality" means two or more, unless otherwise specifically defined.

[0136] Although the present application is described herein in conjunction with various embodiments, in the process of implementing the claimed application, those skilled in the art can understand and implement other changes to the disclosed embodiments by reviewing the drawings, the disclosure, and the appended claims. In the claims, the word "comprising" does not exclude other components or steps, and "a" or "an" does not exclude multiple situations. A single processor or other unit can implement several functions listed in the claims. Certain measures are recorded in different dependent claims, but this does not mean that these measures cannot be combined to produce good results.

[0137] The above is a further detailed description of the present invention in conjunction with specific preferred embodiments, and the specific implementation of the present invention should not be considered to be limited to these descriptions. For those skilled in the art of the present invention, without departing from the concept of the present invention, several simple deductions or substitutions can be made, which should be considered to fall within the scope of protection of the present invention.

Claims

1. A face photo-sketch synthesis method based on two-way conditional normalization, characterized in that: include: Obtain a face photo-sketch pair to be synthesized; Inputting the face photo-sketch pair to be synthesized into a trained face photo-sketch synthesis network to obtain a synthesis result; The face photo-sketch synthesis network includes an encoder, a generator, and a semantic segmentation module. The generator includes a two-way normalization module, a gated attention feature fusion module, and a decoder. The trained face photo-sketch synthesis network is obtained by training a training set of face photo-sketch pairs. The corresponding training process includes: The encoder encodes the face photo-sketch pair training set and outputs deep features; the semantic segmentation module extracts semantic labels of the sketches in the training set; the two-way normalization module strengthens the spatial information branch and the texture information branch according to the semantic labels and the deep features; the gated attention feature fusion module fuses the output results of the spatial information branch and the texture information branch; the decoder decodes the fusion result and outputs the synthesis result corresponding to the face photo-sketch pair training set; the loss function of the face photo-sketch synthesis network is constructed according to the face photo-sketch pair training set and its corresponding synthesis result; the parameters of the face photo-sketch synthesis network are updated according to the loss function, and training is continued until the iteration stop condition is met to obtain the trained face photo-sketch synthesis network.

2. The face photo-sketch image synthesis method based on two-way conditional normalization according to claim 1 is characterized in that: The encoder includes a plurality of convolutional layers connected in sequence; In the encoder, the outputs corresponding to the last three convolutional layers of the encoder are used as deep features.

3. The face photo-sketch image synthesis method based on two-way conditional normalization according to claim 1 is characterized in that: The dual-path normalization module includes a SPADE Resblock module and an AdaIN Resblock module, wherein: The SPADE Resblock module enhances spatial information based on the semantic labels of the sketches in the training set and the depth features of the face photos in the training set; The AdaIN Resblock module enhances texture information based on the depth features of sketches and face photos in the training set.

4. The face photo-sketch image synthesis method based on two-way conditional normalization according to claim 3 is characterized in that: The AdaIN Resblock module includes a plurality of sequentially connected residual AdaIN modules; wherein each of the residual AdaIN modules includes a basic AdaIN module and a residual module that are sequentially connected.

5. The face photo-sketch image synthesis method based on two-way conditional normalization according to claim 3 is characterized in that: The output of the SPADE Resblock module is expressed as: Among them, p represents a face photo, A i represents the output of the SPADE Resblock module corresponding to the i-th (i=1,2,3) layer depth feature, represents the depth feature corresponding to the face photo p in the i-th (i=1,2,3) layer depth feature, c represents the channel, (y,x) represents At a position on channel c, express The value at point (y,x) on channel c, M s represents a semantic label, Respectively On the c-th channel, from the semantic label M s The scaling and offset learned in Respectively Mean and variance on the c-th channel.

6. The face photo-sketch image synthesis method based on two-way conditional normalization according to claim 5, characterized in that: The output of the AdaIN Resblock module is expressed as: Among them, s represents a sketch, B i represents the output of the AdaIN Resblock module corresponding to the i-th (i=1,2,3) layer depth feature, represents the depth feature corresponding to the sketch image s in the i-th (i=1,2,3) layer depth feature, express On channel c The corresponding value at the same point (y,x), Respectively Mean and variance on the c-th channel.

7. The face photo-sketch image synthesis method based on two-way conditional normalization according to claim 6, characterized in that: The gated attention feature fusion module includes two gate modules and a channel attention module; wherein, The outputs of the SPADE Resblock module and the AdaIN Resblock module are respectively input into two gating modules; The outputs of the SPADE Resblock module and the AdaIN Resblock module are superimposed and input into the channel attention module; The outputs of the two gating modules are fused with the output of the channel attention module.

8. The face photo-sketch image synthesis method based on two-way conditional normalization according to claim 7, characterized in that: The output of the two gating modules and the output of the channel attention module are fused and expressed as: Among them, C i represents the output of the gated attention feature fusion module corresponding to the depth feature of the i-th layer (i=1, 2, 3), CA(·) represents the channel attention function, Indicates A i The corresponding gating function, Indicates B i The corresponding gating function.

9. The face photo-sketch image synthesis method based on two-way conditional normalization according to claim 1, characterized in that: The decoder is a decoder based on the AFF module; In the decoder, after upsampling the output of the gated attention feature fusion module to obtain features of the same resolution, the AFF module is used to perform feature fusion and decode the output.

10. The face photo-sketch image synthesis method based on two-way conditional normalization according to claim 1, characterized in that: The loss function of the constructed face photo-sketch synthesis network is expressed as: L full =λ1L adversarial +λ2L cycle +λ3L perceptual ; Among them, L full represents the loss function of the face photo-sketch synthesis network; L adversarial represents the generation adversarial loss, L cycle represents the cycle consistency loss, L perceptual represents the perceptual loss, and λ1, λ2, and λ3 represent the balance parameters.

Images