An example-based unsupervised face image translation method

Abstract

The application discloses an example-based unsupervised face image translation method, comprising the following steps: S1, constructing a data set; S2, constructing two independent encoders, the encoders being a style encoding network S and a content encoding network C respectively; S3, the style encoding network encodes an example style image v to obtain a style feature f s (v), the style encoding network encodes a content image u to obtain a content feature f s (u), S4, performing dense semantic matching on the style feature f s (v) and the semantic feature f c (h) obtained in the step S3, and constructing a rough semantic graph; S5, constructing an image decoding network to obtain a fine face image translation result; and S6, performing identity and style constraint to generate a more fine face generation effect. The method divides the cross-domain face image translation task into two parts based on unsupervised learning, first constructs a rough semantic matching feature, and then refines through a decoding network, so that a fine face translation effect is realized.

Description

TECHNICAL FIELD

[0001] The application belongs to the field of image processing, and particularly relates to an example-based unsupervised face image translation method. BACKGROUND

[0002] Image-to-image translation refers to learning a correlation mapping between the source domain and the target domain. It has achieved success in a wide range of fields, such as image style transfer, image inpainting, face sketch generation, image super-resolution reconstruction, semantic segmentation, and data augmentation, etc. These applications can be applied to digital entertainment and public security fields.

[0003] Traditional image-to-image translation tasks all use digital image processing and signal processing methods, such as brush rendering, image analogy, image filtering, texture synthesis, etc. However, these models generally need a large amount of manual parameter tuning and have the problem of weak expression of deep semantic information of images. Since the neural network image translation technology is proposed, methods are realized through convolution, activation function and nonlinear layer, etc. At present, most image translation models use strictly one-to-one paired training data sets to strengthen the learning of affine transformation of neural networks, and good results are achieved.

[0004] In real application scenarios, most data sets are not paired and annotated, and users want to learn multiple mapping relationships through a model. Therefore, example-based image translation has received extensive attention, which allows users to flexibly control the sample style to generate the desired style image. However, the quality of the generated image is not negligible when the sample style is generated, and a large data size and more abundant resources are needed, thereby making the task more challenging. In fact, convolutional neural networks will encounter the following challenges in the process of cross-domain face image translation:

[0005] Large amount of data, making it necessary to consume a large amount of human resources to produce data sets, and previous neural networks need large data sets to achieve good results;

[0006] Insufficient cross-domain semantic matching, previous semantic matching methods only use attention mechanisms, and the unmatched positions are not given sample semantics, resulting in poor generation effect.

[0007] Poor generalization, most cross-domain image translation methods can only perform well within the data set, and the effect of migrating to external faces is often poor. SUMMARY

[0008] In view of the deficiencies in the prior art, the present application provides an example-based unsupervised face image translation method, which uses an unsupervised learning method to subdivide the cross-domain face image translation task into two parts, first constructs rough semantic matching features, and then refines semantic information, is subjected to style contrast learning constraints, identity constraints and global style injection, and strengthens the model generation effect through a multi-scale discriminator, so as to realize a better face translation effect.

[0009] The present application provides an example-based unsupervised face image translation method, comprising the following steps:

[0010] S1, a data set is constructed, which contains example style images and content images;

[0011] S2, a domain-in semantic alignment and domain-out semantic matching strategy is adopted to construct two independent encoders, namely a style encoding network S and a content encoding network C;

[0012] S3, the style encoding network encodes the example style image v to obtain a style feature f s (v), the content image u is encoded by the content encoding network to obtain a content feature f s (u), and then a HED algorithm is used to extract a semantic graph h from the content image u, and the content encoding network extracts a semantic feature f c (h) of the semantic graph h, and semantic alignment constraints are performed on the semantic feature f c (h) and the content feature f s (u); step S3 is used to alleviate the influence of the content feature on the subsequent generated results; f c (h) does not contain original style information, and it is similar to decoupling the identity and style.

[0013] S4, dense semantic matching is performed on the style feature f s (v) and the semantic feature f c (h) obtained in step S3 to construct a rough semantic graph;

[0014] S5, an image decoding network is constructed, and the rough semantic graph is taken as input, and through multiple feature fusion and style injection of the image decoding network, a fine face image translation result is obtained;

[0015] S6, identity and style constraints are performed on the fine face image translation result to generate a more fine face generation effect.

[0016] As a preferred, the step S1 further comprises preprocessing of the data set: the content image and the example image are cropped and flipped by a data enhancement method.

[0017] As preferred, the implementation method in step S4 is as follows:

[0018] The style feature f s (v) is calculated by c (h) Cosine attention to construct a mask vector M mask (h, v), and the semantic matching is divided into relevant semantic regions and irrelevant semantic regions. In order to distinguish the two, a confidence matrix confmap is constructed, and the expression is as follows:

[0019]

[0020]

[0021] confmap = M(h, v) · M mask (h, v)

[0022] Wherein f s (v) is standardized in the channel dimension, f c (h) is standardized in the channel dimension, and M(h, v) represents the correlation degree between semantics.

[0023] For the relevant semantic region, an attention mechanism is used to learn the fine-grained semantic matching:

[0024]

[0025] Wherein α is the control softmax sharpness coefficient,

[0026] For the irrelevant semantic region, conditional adaptive normalization is used for modulation. First, a convolution layer is used to learn the modulation parameter, and further semantic information is supplemented. The expression is as follows:

[0027] warp coarse = γ × f c (h) + β

[0028] warp fine = (1-confmap) * warp coarse

[0029] Wherein, γ is the weight, and β is the deviation,

[0030] Finally, the semantic information is fused to generate a rough semantic feature, which is as follows:

[0031] warp = Norm(warp atten + warp fine + f c (h))

[0032] wherein Norm is normalization.

[0033] As preferred, the method for constructing the image decoding network in step S5 is as follows:

[0034] The image decoding network is composed of ResBlock and ResAdaIN alternately, and the number of each decoding layer of ResAdaIN is denoted as f i ; a fully connected style mapping network MLP is constructed and the style feature f s (v) obtaining the global style feature s' by decoding through MLP, first, inputting the rough semantic map warp into the image decoding network, and then injecting the style feature s' into the decoding layer f i according to the hierarchical injection manner, so as to maximize the consistency of the generated style with the example style, that is, obtaining the fine face image translation result x.

[0035] As preferred, in step S5, the hierarchical injection manner is as follows:

[0036] f i ′=ResAdΙN(f i ,s′)

[0037]

[0038] wherein μ(·) and σ(·) represent mean and standard deviation respectively.

[0039] As preferred, the fully connected style mapping network MLP is composed of 3 layers of Linear and 2 layers of activation function ReLU.

[0040] As preferred, the specific operation of identity and style constraint on the generated image in step S6 is as follows:

[0041] The HED algorithm is used to extract the edge map of the input content image and the edge map of the generated result, and the LPIPS is used to construct the identity soft constraint:

[0042] L id =LPIPS(Hed(c)),Hed(out)

[0043] The global style contrast loss L style is used to constrain the maximum consistency of the generated image with the example style:

[0044]

[0045] wherein z represents the style feature of the generated result, z + the style semantics of the example image obtained by style encoding and MLP decoupling, z -Other style semantics cached during the representative network training process, τ Temperature hyperparameter.

[0046] The generated semantic graph extraction edge structure graph and the edge structure graph extracted from the content image are calculated image similarity to perform identity constraint, so that the network generated graph identity is more similar to the content image.

[0047] For the discriminator, a multi-scale discriminator is adopted, and a plurality of different resolutions are scaled for the generated image and the content image as the input of the discrimination network, and the discrimination network discriminates the generation quality of a plurality of small blocks randomly cropped from the image, so as to improve the overall generation quality of the network.

[0048] The present application has the following beneficial effects:

[0049] For the cross-domain face image translation problem, the present application proposes an example-based unsupervised image translation method, which adopts a progressive strategy, first performs rough semantic matching, and then further refines the generation quality.

[0050] In view of the problems of data generalization and small quantity, the present application adopts AdaIN global style injection and style contrast constraint to improve the model generalization, so that a large data set is not required to achieve good image translation effect, and the external face effect is obvious. The present application adopts the strategy of intra-domain semantic alignment and inter-domain semantic matching, and provides an effective technical route for example-based face image translation. It realizes excellent generalization ability, and can be used as a practical tool for users to enjoy flexible artistic creation.

[0051] The unsupervised learning method is used to perform semantic matching on the cross-domain image, and then the AdaIN style injection is further used, and the generation effect is further modulated through the style contrast constraint, the identity constraint and the multi-scale discriminator, so that good non-paired cross-domain face image translation effect is realized. BRIEF DESCRIPTION OF DRAWINGS

[0052] Figure 1 It is a schematic diagram of an example-based face cross-domain image translation network

[0053] Figure 2 It is a schematic diagram of an inter-domain semantic matching transformer module

[0054] Figure 3 It is a schematic diagram of style contrast constraint DETAILED DESCRIPTION

[0055] The present application is further described below in combination with the drawings.

[0056] The present embodiment provides an example-based unsupervised face image translation method, as shown in Figure 1 And 2As shown, comprising the following steps:

[0057] S1, constructing a data set, which contains example style images and content images;

[0058] S2, for the challenge of cross-domain face image translation, a strategy of intra-domain semantic alignment and inter-domain semantic matching is adopted, and two independent style encoding networks EncodeS and content encoding networks EncoderC are designed;

[0059] It should be noted that the style encoding network EncodeS and the content encoding network EncoderC are both conventional encoders.

[0060] S3, the style encoding network encodes the example style image v to obtain the style feature f s (v), the style encoding network encodes the content image u to obtain the content feature f s (u), and then the HED algorithm is used to extract the semantic graph h of the content image u, and the content encoding network extracts the semantic feature f c (h) of the semantic graph h, and the semantic alignment constraint is performed on the semantic feature f c (h) and the content feature f s (u);

[0061] S4, dense semantic matching is performed on the style feature f s (v) and the content feature f s (u) obtained in step S3, a dynamic modulation strategy is adopted, and a mask attention transformer module is designed, as shown in Figure 2 Through the cosine attention mechanism, the part with weak semanticity is selected for spatial adaptive modulation (SPADE), and the part with strong semanticity is directly matched with semantics, to obtain a rough semantic graph;

[0062] Specifically, the method is as follows:

[0063] Through the cosine attention of the style feature f s (v) and the semantic feature f c (h), a mask vector M mask (h,v) is constructed, the semantic matching is divided into relevant semantic area and irrelevant semantic area, in order to distinguish the two, a confidence matrix confmap is constructed, and the expression is as follows:

[0064]

[0065]

[0066] confmap=M(h,v)·M mask (h,v)

[0067] wherein is the style feature f s (v) in the channel dimension, M(h, v) represents the correlation degree between semantics; is the semantic feature f c (h) in the channel dimension, M(h, v) represents the correlation degree between semantics;

[0068] For relevant semantic regions, attention mechanism is used to learn fine-grained semantic matching:

[0069]

[0070] wherein a is a control coefficient of softmax sharpness,

[0071] For irrelevant semantic regions, conditional adaptive normalization is used for modulation. First, a convolution layer is used to learn the modulation parameters, and further semantic information is supplemented. The expression is as follows:

[0072] warp coarse = γ × f c (h) + β

[0073] warp fine = (1 - confmap) * warp coarse

[0074] wherein γ is a weight, and β is a bias,

[0075] Finally, semantic information fusion is performed to generate a rough semantic feature, as follows:

[0076] warp = Norm(warp atten + warp fine + f c (h))

[0077] wherein Norm is normalization.

[0078] S5, the rough semantic map obtained in S4 is further modulated, an image decoding network Decoder similar to the content encoding network is designed, which is composed of ResBlocks and ResAdaIN alternately, and each layer of ResAdaIN is denoted as f i ; a fully connected style mapping network MLP is designed, and the style feature f s (v) is decoded through the MLP.

[0079] It can be understood that the style encoding network EncoderS, the content encoding network EncoderC, and the decoding network Decoder in steps S2 and S5 above constitute a complete image translation network in the embodiment.

[0080] Specifically, first, the rough semantic graph warp is input into the image decoding network, and then the style feature s' is injected into the decoding layer f i to maximize the consistency of the generated style with the example style, that is, to obtain a fine face image translation result x.

[0081] Wherein, the hierarchical injection method is as follows:

[0082] f i ′=ResAdΙN(f i ,s′)

[0083]

[0084] Wherein, μ(·) and σ(·) represent the mean and standard deviation respectively.

[0085] For the fully connected style mapping network, it is composed of 3 layers of Linear and 2 layers of activation function ReLU, for style feature extraction.

[0086] S6, for example-based cross-domain semantic image translation, in order to guide the model to generate more fine face generation effect, identity and style constraints are needed.

[0087] The input content image edge map and the generated image edge map are extracted by using Hed algorithm, and the identity soft constraint is constructed by using LPIPS:

[0088] L id =LPIPS(Hed(c)),Hed(out)

[0089] The global style contrast loss L style is used to constrain the maximum consistency of the generated image and the example style:

[0090]

[0091] Wherein, z represents the style feature of the generated result, z + The style semantics of the example image obtained by decoupling EncoderS and MLP, z - represents the other style semantics cached in the network training process, and τ is a temperature hyperparameter;

[0092] Further, the generative adversarial network and the adversarial loss are used to improve the image translation generation quality and constrain the identity information of the generated image:

[0093]

[0094]

[0095] where D denotes a discriminator in the generative adversarial network, h(t) = min(0, -1 + t) is a hinge loss used to regularize the discriminator.

[0096] The generated semantic graph extracts the edge structure graph, and the edge structure graph extracted from the content image is calculated for image similarity to perform identity constraint, so as to make the network generated graph more similar to the content image. A multi-scale discriminator is used for the discriminator, and a plurality of different resolutions are scaled for the generated image and the content image as the input of the discrimination network. The discrimination network discriminates the generated quality of a plurality of small blocks randomly cropped from the image, so as to improve the overall generated quality of the network.

[0097] The above is a further detailed description of the present application in combination with specific / preferred embodiments, and cannot be regarded as limiting the specific implementation of the present application to these descriptions. For ordinary skilled persons in the technical field to which the present application belongs, a number of substitutions or modifications can be made to the described embodiments without departing from the concept of the present application, and these substitutions or modifications shall be regarded as falling within the protection scope of the present application.

[0098] It should be noted that some parts of the present application not described in detail are known to those skilled in the art.

Claims

1. An example-based unsupervised face image translation method, characterized in that, The method comprises the following steps: S1, constructing a data set, wherein the data set contains example style images and content images; S2, constructing two independent encoders using the strategy of intra-domain semantic alignment and inter-domain semantic matching, wherein the encoders are a style encoding network S and a content encoding network C respectively; S3, the style encoding network encodes the example style image v to obtain a style feature f s (v), the content encoding network encodes the content image u to obtain a content feature f s (u), then a HED algorithm is used to extract a semantic graph h from the content image u, and the content encoding network extracts a semantic feature f c (h) of the semantic graph h according to Manhattan distance L1; c (h) and the content feature f s (u) are semantically aligned. S4. Deriving style features f from the style features f obtained in step S3 s (v) and semantic features f c (h) performing dense semantic matching, constructing a coarse semantic map; S5, constructing an image decoding network, inputting a rough semantic map into the image decoding network, and obtaining a fine face image translation result through multiple feature fusion and style injection of the image decoding network; The construction method of the image decoding network is as follows: The image decoding network is composed of ResBlock and ResAdaIN alternately, and the number of each decoding layer of ResAdaIN is f i ; a fully connected style mapping network MLP is constructed, and style features f s (v) obtaining global style features s' by decoding through the MLP, first, inputting the rough semantic graph warp into the image decoding network, and then injecting the style features s' into the decoding layer f i according to the hierarchical injection mode, so that the generated style is maximally consistent with the example style, that is, a fine face image translation result x is obtained. The fully connected style mapping network MLP is composed of 3 layers of Linear and 2 layers of activation function ReLU. S6, generating a more fine face generation effect by identity and style constraint for the fine face image translation result, and performing identity and style constraint.

2. The unsupervised face image translation method based on examples according to claim 1, characterized in that, The step S1 further comprises preprocessing of the data set: using a data enhancement method to crop and flip the content images and example images. 3.The unsupervised face image translation method based on examples of claim 1, wherein, The implementation method in the step S4 is as follows: By calculating the style feature f s (v) and the semantic feature f c (h) the cosine attention to construct the mask vector M mask (h, v), the semantic matching is divided into relevant semantic regions and irrelevant semantic regions, in order to distinguish the two, a confidence matrix confmap is constructed, and the expression is as follows: confmap = M(h, v) · M mask (h, v) wherein is the style feature f s (v) in the channel dimension, M(h, v) represents the degree of correlation between semantics; is the semantic feature f c (h) in the channel dimension, M(h, v) represents the degree of correlation between semantics; For related semantic regions, an attention mechanism is used to learn fine-grained semantic matching: Wherein α is a control softmax sharpness coefficient, For unrelated semantic regions, conditional adaptive normalization is used for modulation, a convolution layer is used to learn modulation parameters, and further semantic information is supplemented, and the expression is as follows: warp coarse = γ x f c (h) + β warp fine = (1 - confmap) * warp coarse Wherein γ is a weight and β is a bias, Finally, the semantic information is fused to generate a rough semantic feature, and the specific process is as follows: warp = Norm(warp atten + warp fine + f c (h)) Wherein, Norm is normalization.

4. The unsupervised face image translation method based on examples according to claim 1, characterized in that, In the step S5, the hierarchical injection method is as follows: Wherein μ(·) and σ(·) represent mean and standard deviation respectively.

5. The unsupervised face image translation method based on examples according to claim 1, characterized in that, The specific operation of identity and style constraint on the generated image in the step S6 is as follows: The HED algorithm is used to extract the edge map of the input content image and the edge map of the generated result, and the LPIPS is used to construct an identity soft constraint: L id = LPIPS(Hed(c), Hed(out)) using a global style contrast loss L style to constrain the generated image to be maximally consistent with the example style: where z represents the style characteristics of the generated results, z + Example images with style semantics decoupled by style encoding and MLP, - represent other style semantics cached during the network training process, τ is the temperature hyperparameter.

Images