Face cross-domain translation method based on multi-view knowledge integration and frequency consistency

Abstract

The application discloses a face cross-domain translation method based on multi-view knowledge integration and frequency consistency, which introduces face knowledge in large single-domain data into limited cross-domain data. Specifically, after the input photo is extracted by a face analysis network, the photo is cascaded into a translation network based on a multi-view knowledge integration model. On the other hand, the input photo is extracted by a face recognition network to obtain face identity features, and the cascaded features and the face identity features are used to calculate KL loss, thereby identity consistency supervision is achieved to enhance the high and low frequency consistency between the generated image and the real image. The frequency consistency loss is a general reconstruction loss, which can be flexibly applied to other generation models to improve the robustness of the model to noise.

Description

TECHNICAL FIELD

[0001] The present application belongs to the technical field of computer vision, and particularly relates to a face cross-domain translation method based on multi-view knowledge integration and frequency consistency. BACKGROUND

[0002] The purpose of face cross-domain translation is to translate face images between different domains, including face photo and sketch, photo and line drawing, and near-infrared (NIR) and visible light (VIS). It can be applied to various scenarios, from law enforcement to digital entertainment, and has recently attracted widespread attention. Artistic portrait drawings (APDrawings) are composed of sparse lines, which are used to construct rough structural information of faces. In addition, face photo-sketch / photo-line drawing translation has also been proven to improve the efficiency of animation production, or provide personalized avatars for people without professional drawing training on social platforms. In addition, NIR-VIS face image translation is widely used to improve the security of access control systems. Near-infrared cameras capture reflected near-infrared light, which is not affected by visible light conditions, while visible light cameras are deeply dependent on light conditions, and in low light or dark conditions, it is easy to produce low-quality images with blurred textures. However, due to the existence of obvious style domain differences, it is difficult to directly match near-infrared images with images in a citizen face database. Face NIR-VIS translation can make up for this deficiency and improve the accuracy of face recognition.

[0003] Methods for cross-domain face translation can be divided into two categories: sample-based methods and GAN-based methods. Sample-based methods increase training data by cropping face images into image patches and simplifying image-level translation tasks to image patch-level. The output image is reconstructed by weighting image patches with a set of reconstruction coefficients. However, these methods are time-consuming and can produce blurred or over-smoothed structures. RSLCR accelerates the face photo-sketch synthesis process by random sampling and local constraints. DPGM combines PGM and deep discriminative patch representation, which jointly models deep patch representation and sketch patch reconstruction, but this type of method still needs more time to search for the nearest neighbor image patch and optimize the reconstruction coefficients.

[0004] The significant progress of convolutional neural networks has inspired recent GAN-based approaches that mainly directly translate input face images from the source domain to the target domain and require sufficient samples to produce realistic textures.Pix2pix is a conditional GAN model that learns image-to-image mapping and achieves considerable performance in image translation tasks.KT uses a pre-trained teacher network to guide the generation network.However, limited by complex imaging conditions and costs, paired cross-domain face images are difficult to obtain, resulting in limited cross-domain data scales.In view of this limitation, even if the synthesized appearance is usually realistic, these methods often produce unreasonable textures or severe deformations, especially in the eye and hair regions. SUMMARY

[0005] The purpose of the present application is to overcome the deficiencies in the prior art and provide a face cross-domain translation method based on multi-view knowledge integration and frequency consistency, which combines information of different views by introducing pre-trained structure modules and identity modules to improve the structural stability of generated images and maintain consistent identities; the frequency consistency loss containing multi-direction Prewitt loss and Gaussian blur loss can simultaneously constrain the generated images to have clearer structures, realistic textures and reasonable color distributions in the high-frequency domain and the low-frequency domain.

[0006] The purpose of the present application is realized by the following technical solutions:

[0007] A face cross-domain translation method based on multi-view knowledge integration and frequency consistency, the multi-view knowledge integration model includes a structure module, an identity module and a translation network T; the translation network T includes two mappings between the style domain X and the style domain Y, namely Ty: X→Y and Tx: Y→X; the translation network T adopts an encoder-decoder architecture with 9 residual blocks as the backbone network; in the training stage, by representing the basic generation constraint of mapping images from the source domain to the target domain:

[0008]

[0009] where x and y represent a sample in the style domain X and the style domain Y respectively, generated by the translation network T y input x, while generated by the translation network T x input y; in addition, the cycle consistency loss excites x rec =x,y rec =y, where x rec is generated by the translation network T x input , y rec is generated by the translation network Ty Input Generate:

[0010]

[0011] In the multi-view knowledge integration model, two discriminators D x and D y serve the translation network T y and the translation network T x respectively, the adversarial loss is represented as:

[0012]

[0013] Where represents expectation, T y tries to generate more realistic images T y (x), D y focuses on distinguishing between real images y and generated images T y (x); accordingly, T x tries to generate more realistic images T x (y), D x focuses on distinguishing between real images x and generated images T x (y) Adversarial loss in the opposite translation direction is represented as:

[0014]

[0015] The complete adversarial loss is:

[0016]

[0017] Finally, the goal of the translation network is represented as follows:

[0018]

[0019] The structure module uses a pre-trained face parsing network to obtain a face component mask as prior knowledge, and trains a self-encoder from the face component mask to the target domain image, and obtains the cascaded feature f S by cascading the structure knowledge f T and the feature f S,T of the translation network, the structure knowledge f S includes and the feature f T of the translation network includes and f S,T includes and The structure module can provide structural knowledge to the translation network to reduce confusion noise between different face components and suppress outlier pixel values within each face component;

[0020] The identity module is implemented by a pre-trained face recognition network, and the loss function of the multi-view knowledge integration model is set by setting the deviation of the distribution of the cascaded features and the distribution of the identity features is expressed as:

[0021]

[0022] where f I represents the knowledge of the identity view, P and Q are and The expected distribution at the i-th knowledge aggregation;

[0023] The frequency consistency loss is obtained by calculating the multi-direction mPrewitt loss and the Gaussian blur loss of the translation network T, which respectively constrains the high frequency and low frequency consistency.

[0024] Further, the frequency consistency auxiliary generation constraint is introduced and the generated image in the frequency domain is constrained by the frequency consistency loss; the frequency consistency loss is composed of multi-direction Prewitt loss and Gaussian blur loss, as follows:

[0025] (1) Multi-direction Prewitt loss;

[0026] The multi-direction Prewitt operator is applied as a predefined convolution kernel, and the gradient images in four directions of the input image, i.e., horizontal, vertical, main diagonal and secondary diagonal, are obtained. The Prewitt operator calculates the gradient of each pixel and its surrounding 8 pixels:

[0027]

[0028] where W i represents the i-th kernel of W, represents the i-th gradient image of , and represents the generated image; the positive value of the gradient image indicates that the gradient around the corresponding pixel is the same as the direction of the mPrewitt operator, and the negative value indicates that the direction is opposite; zero indicates no gradient change; by constraining the generated image and and the real image and between the L1 norm:

[0029]

[0030] (2) Gaussian blur loss;

[0031] The Gaussian blur kernel can blur the content of the image and expose the style of the image, and a k x k size Gaussian blur kernel is generated by the following formula:

[0032]

[0033]

[0034] Where m, n represents the (m, n) position with the center of the kernel as the origin; sigma represents the standard deviation of the Gaussian function, and the sum of the Gaussian blur kernel is kept to be 1 by normalization to obtain K', the blurred image y b The image y is calculated by the following formula:

[0035] y b = K' * y (12) Gaussian blur loss The blurred version of the generated image is constrained by the L1 norm And The low-frequency consistency between the template version y b And x b , defined as:

[0036]

[0037] In summary, the total loss function of the multi-view knowledge integration model Is:

[0038]

[0039] Where, Indicates the basic generation constraint of mapping the image from the source domain to the target domain, Indicates the cycle consistency loss, Is the adversarial loss.

[0040] The application also provides an electronic device, including a memory, a processor and a computer program stored on the memory and executable on the processor, wherein the processor implements the steps of the face cross-domain translation method based on multi-view knowledge integration and frequency consistency when executing the program.

[0041] The application also provides a computer readable storage medium having a computer program stored thereon, wherein the computer program is executed by a processor to implement the steps of the face cross-domain translation method based on multi-view knowledge integration and frequency consistency.

[0042] Compared with the prior art, the technical scheme of the application has the beneficial effects that:

[0043] 1. The present application proposes a multi-view knowledge integration model for cross-domain face translation. Face knowledge in large single-domain data is introduced into limited cross-domain data. Specifically, external structural knowledge from a face semantic segmentation model and identity knowledge from a face recognition model are integrated into the translation network through a knowledge aggregation module.

[0044] 2. To better integrate multi-view knowledge, a novel frequency consistency loss is further designed to enhance the high and low frequency consistency between the generated image and the real image. The proposed frequency consistency loss is a general reconstruction loss that can be flexibly applied to generative adversarial networks, variational autoencoders and other deep generative models. It can effectively encourage the model to learn the continuous change pattern between pixels and the color gamut of real images, and improve the model's robustness to noise.

[0045] 3. Through sufficient experimental comparison and ablation experiments, it is verified that the introduction of large-scale pre-trained models and improvements in the frequency domain can improve the performance of the image translation model. BRIEF DESCRIPTION OF DRAWINGS

[0046] Figure 1 is a schematic diagram of the multi-view knowledge integration model proposed in the specific embodiment.

[0047] Figure 2 is an effect diagram of the multi-directional gradient operator extracting the gradient map introduced by the present application. DETAILED DESCRIPTION

[0048] The present application will be further described in detail below in combination with the drawings and specific embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and do not limit the present application.

[0049] See Figure 1 To achieve cross-domain face translation, the present embodiment provides a multi-view knowledge integration and frequency consistency model. The multi-view knowledge integration model is composed of three parts: a structure module (S), an identity module (I) and a translation network (T). In addition, during the training phase, the frequency consistency loss is calculated by calculating the mPrewitt loss and the Gaussian blur loss of the translation network, respectively, to constrain the high and low frequency consistency.

[0050] I. Multi-view knowledge integration model

[0051] 1. Translation network

[0052] The translation network T includes two mappings between the style domain X and the style domain Y, i.e. T y : X→Y and T x : Y→X. T adopts an encoder-decoder architecture with 9 residual blocks as the backbone network. During the training phase, the frequency consistency loss is calculated by The basic generation constraint that represents mapping images from the source domain to the target domain:

[0053]

[0054] In this embodiment, x and y represent a sample in the photo domain and sketch domain, respectively, generated by the translation network T y from input x generated by the translation network T x from input y. In addition, the cycle-consistency loss is also encouraged for x rec = x, y rec = y, where x rec is generated by the translation network T x from input y rec is generated by the translation network T y from input :

[0055]

[0056] In the multi-view knowledge integration model, two discriminators D x and D y serve T y and T x respectively, and the adversarial loss can be represented as:

[0057]

[0058] where represents expectation, T y tries to generate more realistic images T y (x), and D y focuses on distinguishing real images and generated images. Accordingly, T x tries to generate more realistic images T x (y), and D x focuses on distinguishing real images x and generated images T x (y). The adversarial loss of the opposite translation direction can be represented as:

[0059]

[0060] The complete adversarial loss is:

[0061]

[0062] Finally, the goal of the translation network is represented as follows:

[0063]

[0064] 2. Structure module

[0065] The semantic segmentation framework can be used to parse the face components, which can be used to represent a face image from the structure view. Therefore, the structure module of the embodiment utilizes a pre-trained face parsing network to obtain a face component mask as prior knowledge, and trains an autoencoder from the face component mask to the target domain image, thereby obtaining the cascaded structure knowledge f S (including and ) and the features f T (including and ) of the translation network to obtain the cascaded features f S,T (including and ). The structure module can provide the structure knowledge to the translation network. This process can significantly reduce the confusion noise between different face components and suppress the outlier pixel values within each face component.

[0066] 3. Identity module

[0067] The identity module is implemented by a pre-trained face recognition network. The embodiment attempts to integrate the knowledge f I (including and ) of the identity view into the cascaded features f S,T . Considering that the face recognition task and the image translation task are quite different, and the identity consistency needs to be maintained, a relatively weak supervision KL divergence is set to measure the deviation of the distribution of the cascaded features and the distribution of the identity features. The multi-view knowledge integration loss function can be represented as:

[0068]

[0069] where P and Q are the expected distributions of and at the i-th knowledge aggregation.

[0070] II. Frequency consistency loss

[0071] Integrating multi-view knowledge into the translation task can increase effective information, but also introduce redundant information. Here, frequency consistency is introduced to assist the traditional generation constraint, and a frequency consistency loss is proposed to constrain the generated images in the frequency domain. It refines more effective information for the image translation task. The proposed frequency consistency loss consists of a multi-direction Prewitt loss and a Gaussian blur loss.

[0072] 1. Multi-directional Prewitt loss

[0073] Multi-directional Prewitt loss function: Apply multi-directional Prewitt operator as predefined convolution kernel and get the gradient maps of input image in four directions (horizontal, vertical, major diagonal, minor diagonal), Prewitt operator computes gradient by each pixel and its 8 surrounding pixels:

[0074]

[0075] where W i represents the i-th kernel of W, represents the i-th gradient map of and represents the generated image. Positive values of gradient map indicate the gradient around this pixel is in the same direction as the mPrewitt operator, while negative values indicate the opposite direction. Zero indicates no gradient change. The proposed constrains the generated image and to be close to the real image and in L1 norm:

[0076]

[0077] 2. Gaussian blur loss

[0078] Gaussian blur kernel can blur the content of image, expose the style of image, a larger kernel produces more severe blur effect, a k x k size of Gaussian blur kernel can be generated by:

[0079]

[0080]

[0081] where m, n represents the (m, n) position with the center of the kernel as the origin. σ represents the standard deviation of Gaussian function, which affects the weight distribution of the kernel. By normalizing to keep the sum of Gaussian blur kernel as 1 to obtain K', the blurred image y b can be obtained by:

[0082] y b = K' * y (12) Gaussian blur loss constrains the blurred version of the generated image and to be consistent with the template version of the real image y b and x b in low frequency, which can be defined as:

[0083]

[0084] In summary, the total loss function of the multi-view knowledge integration model for:

[0085]

[0086] Specifically, this embodiment uses the CUFS face photo-sketch image dataset for network training and testing. This dataset includes the Chinese University of Hong Kong student database (188 individuals), the AR face database (123 individuals), and the XM2VTS database (295 individuals). Artists created sketches for each individual, resulting in 606 face photo-sketch pairs. For the Chinese University of Hong Kong student database, this embodiment selected 88 identities as the training set and the remaining 100 face photo-sketch pairs as the test set. For the AR face database, 80 face photo-sketch pairs were selected as the training set, and the remaining pairs were used as the test set. For the XM2VTS database, 100 pairs were selected for training, and the rest were used for testing. All face photo-sketch images used in the experiment were aligned with five facial points and cropped to a size of 200×250. The input images were enlarged to 286×286 using zero-padding and then randomly cropped to 256×256.

[0087] Based on extensive experiments, a 3×3 mPrewitt operator and an 11×11 Gaussian blur kernel were ultimately selected. The Adam optimizer was used with momentum parameters β1 = 0.5 and β2 = 0.999, and a learning rate of 0.0002. In the experiments, both the mPrewitt operator and the Gaussian blur kernel were slid with a stride of 1. All experiments were conducted in a Python 3.7.3 and PyTorch 1.7.1 environment on an Ubuntu 18.04 system with a Pascal Titan Xp GPU.

[0088] The multi-view knowledge integration model of this embodiment is trained using the training set described above. The network input is a pair of cross-domain face images. Taking photo-sketch as an example, the input photo extracts features through a face parsing network and then cascades them to a translation network. The other input photo extracts facial identity features through a face recognition network. The cascaded features and facial identity features are used to calculate the KL loss, thereby achieving identity consistency supervision. In addition, the frequency consistency loss further optimizes the multi-view knowledge integration model at high and low frequencies. The detailed network structure is shown in Table 1.

[0089] Table 1 Network Structure

[0090]

[0091]

[0092] After obtaining the trained multi-view knowledge integration model, test is performed on the test set, and the results obtained in the embodiment are compared with the latest baseline models LLE, MRF, MWF, SSD, Bayesian, RSLCR, DPGM, Pix2pix, CycleGAN, PS2MAN and KT in terms of face recognition rate FR, learning perceptual image block similarity LPIPS and Fréchet Inception Distance (FID), and the experimental results are shown in Table 2:

[0093] Table 2 Comparative experiment

[0094]

[0095]

[0096] In the table, the arrowhead indicates that the greater the value of the corresponding index is, the better, and the arrowhead indicates that the smaller the value of the corresponding index is, the better.

[0097] It can be seen from the table that the method proposed in the embodiment has a greater improvement in the index than the previous method, and it is shown that the method proposed in the embodiment can achieve a strong cross-domain face translation effect.

[0098] The present application is not limited to the embodiments described above. The above description of specific embodiments is intended to describe and illustrate the technical solutions of the present application, and the specific embodiments described above are merely illustrative and not restrictive. Without departing from the purpose of the present application and the scope protected by the claims, those skilled in the art can make many forms of specific changes under the inspiration of the present application, and these all belong to the protection scope of the present application.

Claims

1. A cross-domain face translation method based on multi-view knowledge integration and frequency consistency, characterized in that, The multi-view knowledge integration model includes a structural module, an identity module, and a translation network T; the translation network T includes two mappings, Ti, between style domain X and style domain Y. y X→Y and T x The translation network T, Y→X, employs an encoder-decoder architecture with 9 residual blocks as the backbone. During training, it uses... This represents the basic generative constraints for mapping an image from the source domain to the target domain: Where x and y represent a sample from style domain X and style domain Y, respectively. Translated by T network y The input x generates the value, and Translated by T network x Input y is generated; furthermore, through cycle consistency loss. Incentive x rec =x,y rec = y, where x rec It is translated by the translation network T x enter Generate, y rec It is translated by the translation network T y enter generate: In the multi-view knowledge integration model, two discriminators D x and D y Each serves the translation network T y and translation network T x Combating losses Represented as: in To express expectations, T y Attempt to generate more realistic images T y (x), D y Focus on distinguishing between the real image y and the generated image T y (x); correspondingly, T x Attempt to generate more realistic images T x (y), D x Focus on distinguishing between real image x and generated image T x (y) Adversarial loss in the opposite translation direction Represented as: Complete combat losses for: Ultimately, the goal of the translation network will be... It is expressed as follows: The structural module utilizes a pre-trained face parsing network to obtain a face component mask as prior knowledge, and trains an autoencoder from the face component mask to the target domain image, through cascaded structural knowledge f S and the characteristics of translation networks f T The cascaded features f are obtained S,T The cascaded structure knowledge f S include and Features of translation networks f T include and f S,T include and The structural module can provide structural knowledge to the translation network to reduce the confusion noise between different face parts and suppress outlier pixel values ​​within each face part; The identity module is implemented by a pre-trained face recognition network. The loss function of the multi-view knowledge integration model is determined by setting a supervised KL divergence metric to measure the deviation between the distribution of the concatenated features and the distribution of the identity features. Represented as: Among them, f I The knowledge represents the identity view, where P and Q are respectively and The expected distribution at the i-th knowledge aggregation point; Frequency consistency loss is achieved by calculating the multidirectional mPrewitt loss and Gaussian blur loss of the translation network T, which constrain high-frequency and low-frequency consistency respectively.

2. The face cross-domain translation method based on multi-view knowledge integration and frequency consistency according to claim 1, characterized in that, Introducing frequency consistency to assist in generating constraints The generated image in the frequency domain is constrained by a frequency consistency loss, which consists of a multi-directional Prewitt loss and a Gaussian blur loss, as detailed below: (1) Multidirectional Prewitt loss; The multi-directional Prewitt operator is applied as a predefined convolution kernel to obtain gradient maps in four directions: horizontal, vertical, main diagonal, and secondary diagonal of the input image. The Prewitt operator calculates the gradient for each pixel and its eight surrounding pixels. Where M i This represents the i-th core of W. express The i-th gradient map, and Represents the generated image; gradient map Positive values ​​indicate that the gradient around the corresponding pixel is in the same direction as the mPrewitt operator, while negative values ​​indicate the opposite direction; zero indicates no gradient change; through Constraint-generated image and and real images and L1 norm between: (2) Gaussian blur loss; A Gaussian blur kernel can blur the content of an image, revealing its style. A k×k Gaussian blur kernel is generated by the following formula: Where m,n represents the (m,n) position with the center of the kernel as the origin; σ represents the standard deviation of the Gaussian function, which is obtained by normalizing the sum of the Gaussian blur kernels to 1 to obtain K′, and the blurred image y b The image y is obtained by calculating the following formula: and b =K′*y (12) Gaussian blur loss A blurred version of the image is generated using L1 norm constraints. and Template version of the real image y b and x b The low-frequency consistency between them is defined as: In summary, the total loss function of the multi-view knowledge integration model for: in, This represents the basic generative constraints for mapping an image from the source domain to the target domain. Indicates the loss of cycle consistency. To combat the losses.

3. An electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the steps of the cross-domain face translation method based on multi-view knowledge integration and frequency consistency as described in any one of claims 1 and 2.

4. A computer-readable storage medium having a computer program stored thereon, characterized in that, When executed by a processor, the computer program implements the steps of the cross-domain face translation method based on multi-view knowledge integration and frequency consistency as described in any one of claims 1 and 2.

Images