A small sample heterogeneous face conversion method based on collaborative contrast loss
By using a small sample conversion method of collaborative contrast loss in heterogeneous face recognition, unsupervised training is used to solve the diversity and identity consistency of heterogeneous face images, and the CASIA NIR-VIS 2.0 is realized Good performance on the data set of CUHK students surpasses the existing technology.
Patent Information
- Application Number
- CN202210258673.1
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2022-03-16
- Publication Date
- 2024-11-26
- Estimated Expiration
- 2042-03-16
AI Technical Summary
The prior art is difficult to process images from different environments and light rays, especially face images in low-light areas, in the prior art, and face recognition models trained on the visible domain face image are difficult to recognize heterogeneous faces. Moreover, there are problems of diversity and identity consistency in heterogeneous face transformation under small sample learning.
A small sample heterogeneous face conversion method based on collaborative contrast loss is proposed. By acquiring source images, invisible factors and negative samples, the perceived feature extraction model and encoder are used to perform structural and hidden space contrast constraints, and a heterogeneity with consistent identity is generated Face images are trained unsupervised using the CoConGAN framework to ensure the diversity of images and identity consistency.
On CASIA NIR-VIS 2.0 and CUHK datasets, CoConGAN successfully generated heterogeneous face images with high diversity and consistent identity, demonstrating its good generalization performance and visual quality, surpassing existing methods such as DivCo and DVGFace.
Smart Images

Figure CN115147260B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of image processing, and in particular to a small sample heterogeneous face conversion method based on collaborative contrast loss. Background Art
[0002] Heterogeneous face images taken from different environments and lighting conditions can significantly change facial appearance and can cause large variations in the properties of the same person's face. However, face recognition models trained on visible domain face images have difficulty recognizing faces from low-light areas, which poses a great challenge to heterogeneous face recognition (HFR). The goal of heterogeneous face transfer is to synthesize similar face images that retain the identity information of the source domain input image and introduce the style of the target domain reference image. Although current methods require access to images with the same identity information in the source and target domains during training and have achieved great success, we believe that this greatly limits the application of heterogeneous face transfer.
[0003] Given the recent breakthroughs in GANs in image synthesis, face-to-face translation has been widely used, including cross-modal expression generation, facial style transfer, age editing, etc. The core feature of the above methods is to retain all the features in the target face while referencing it. This is exactly what is needed for heterogeneous face translation (HFT) to bridge the domain gap. Although these methods can generate heterogeneous face images that are photo-realistic and identity-preserving (preserving all features in the face), even though the proposed DVG and DVGFace can enrich identity diversity with identity information, they all have a common premise: the training set preserves heterogeneous face images with identity matching, from Figure 1 As can be seen in the left half of the figure, faces with different poses, expressions, and views can be identified as the same person. In applications, it is difficult to determine the same person from different acquisition settings. And it is completely impossible to generalize the results from several face images with unclear identities. When humans see many different objects, it is easy to imagine different visual angles of a new object. Therefore, we propose a face-to-face translation framework for few-shot learning, which aims to learn an HFT model that maps face images in the source domain to similar images with the same identity features by utilizing a few face images of foreign identities. The model has never been used with images of the same identity in the target domain during training, but is required to synthesize them during training and generalize at test time, such as Figure 1 As shown on the right. However, there are two problems that need to be solved based on few-shot HFT: 1) diverse heterogeneous face synthesis, and 2) identity consistency of synthesized images.
[0004] Therefore, it is an urgent problem for those skilled in the art to propose a small sample heterogeneous face conversion method based on collaborative contrast loss to solve the problems existing in the prior art. Summary of the invention
[0005] In view of this, the present invention provides a small sample heterogeneous face conversion method based on collaborative contrast loss to ensure the diversity and identity consistency of heterogeneous face images.
[0006] In order to achieve the above object, the present invention adopts the following technical solution:
[0007] A small sample heterogeneous face conversion method based on collaborative contrast loss, comprising the following steps:
[0008] Obtain an image, a hidden factor, a positive sample and N negative samples, and use the obtained image as the source image;
[0009] Input the source image, an invisible factor, a positive sample and N negative samples into the trained generator, and output the first image group and the second image group with the same identity;
[0010] The perceptual features of the source image and the first image group are extracted respectively by using the perceptual feature extraction model, and feature space correlation graphs of the source image and the first image group are obtained, and used as structural contrast constraints;
[0011] Encode the first image group and the second image group using an encoder, and embed the obtained code into features of an image, features of a positive sample image, and features of N negative sample images as latent space contrast constraints;
[0012] The final converted face image is obtained by using structural contrast constraints and latent space contrast constraints.
[0013] Optionally, the training process of the generator is: inputting a source image and a hidden factor, and outputting a trained image, wherein the hidden factor obeys a normal distribution.
[0014] Optionally, the total loss of the generator is calculated as follows:
[0015]
[0016] Among them, L adv (G,D) is the adversarial loss function, L Lacontra (G) is the latent contrast regularization function, L Stcontra (G) is the structural contrast regularization function; γ is the hyperparameter, G is the generator, and D is the discriminator.
[0017] Optionally, the adversarial loss of the generator is calculated as follows:
[0018] Ladv (G,D)=E[log(D(y))]+E[log(1-D(x,z))] (2)
[0019] Among them, x is the input image from the source domain, y is the reference image from the target domain, and z is the invisible factor.
[0020] Optionally, the cooperative contrastive loss of the generator includes latent contrastive regularization loss and structural contrastive regularization loss.
[0021] Optionally, the latent contrast regularization loss is calculated as follows:
[0022]
[0023] Among them, τ is the scale used to control the size of the loss, e is the feature of the image, and e + is the feature of the positive sample image.
[0024] Optionally, the structural contrast regularization loss is calculated as follows:
[0025]
[0026] in, is the structural loss of the i-th point, N st is the number of points.
[0027] Optionally, the calculation formula for the structured loss of the i-th point is as follows:
[0028]
[0029] in, For and The cosine similarity of For x i The vector of for is a vector of , ζ is the scale factor.
[0030] Optionally, the acquired images are from the CASIA NIR-VIS 2.0 upper and middle school student datasets.
[0031] It can be seen from the above technical solutions that, compared with the prior art, the present invention provides a small-sample heterogeneous face conversion method based on collaborative contrast loss: for the first time, a small-sample unsupervised heterogeneous face conversion framework called CoConGAN is proposed to remove the limitation of identity matching between heterogeneous face images (without pairing data); a collaborative contrast loss algorithm is proposed to jointly ensure the diversity and identity consistency of heterogeneous face images; a large number of experiments conducted on CASIA NIR-VIS 2.0 and the Chinese university student dataset show that CoConGAN has achieved good results and proved that our method has good generalization performance. BRIEF DESCRIPTION OF THE DRAWINGS
[0032] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying creative work.
[0033] Figure 1 Comparison images for heterogeneous face generation based on the same identity and small sample diverse heterogeneous face generation;
[0034] Figure 2 A flow chart of a small sample heterogeneous face conversion method based on collaborative contrast loss provided by the present invention;
[0035] Figure 3 A framework CoConGAN for heterogeneous face conversion provided by the present invention;
[0036] Figure 4 Schematic diagram of the self-similarity comparison of paired heterogeneous face images under the same identity;
[0037] Figure 5 Visual comparison of high-frequency sound quality between CASIA NIR-VIS 2.0 and college students. DETAILED DESCRIPTION
[0038] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.
[0039] Reference Figure 2 As shown, the present invention discloses a small sample heterogeneous face conversion method based on collaborative contrast loss, comprising the following steps:
[0040] Obtain an image, a hidden factor, a positive sample and N negative samples, and use the obtained image as the source image;
[0041] Input the source image, an invisible factor, a positive sample and N negative samples into the trained generator, and output the first image group and the second image group with the same identity;
[0042] The perceptual features of the source image and the first image group are extracted respectively by using the perceptual feature extraction model, and feature space correlation graphs of the source image and the first image group are obtained, and used as structural contrast constraints;
[0043] Encode the first image group and the second image group using an encoder, and embed the obtained code into features of an image, features of a positive sample image, and features of N negative sample images as latent space contrast constraints;
[0044] The final converted face image is obtained by using structural contrast constraints and latent space contrast constraints.
[0045] Optionally, the training process of the generator is: inputting a source image and a hidden factor, and outputting a trained image, wherein the hidden factor obeys a normal distribution.
[0046] Furthermore, the total loss of the generator is calculated as follows:
[0047]
[0048] Among them, L adv (G,D) is the adversarial loss function, L Lacontra (G) is the latent contrast regularization function, L Stcontra (G) is the structural contrast regularization function; γ is the hyperparameter, G is the generator, and D is the discriminator.
[0049] Optionally, the adversarial loss of the generator is calculated as follows:
[0050] L adv (G,D)=E[log(D(y))]+E[log(1-D(x,z))] (2)
[0051] Among them, x is the input image from the source domain, y is the reference image from the target domain, and z is the invisible factor.
[0052] Furthermore, the cooperative contrast loss of the generator includes latent contrast regularization loss and structural contrast regularization loss, which can obtain a large number of new diverse heterogeneous face images with the same identity.
[0053] Optionally, the latent contrast regularization loss is calculated as follows:
[0054]
[0055] Among them, τ is the scale used to control the size of the loss, e is the feature of the image, and e + is the feature of the positive sample image.
[0056] Furthermore, the formula for calculating the structural contrast regularization loss is as follows:
[0057]
[0058] in, is the structural loss of the i-th point, N st is the number of points. This loss function can make corresponding points close and non-corresponding points far away, and can accurately express the structural differences of heterogeneous face images regardless of their visual appearance.
[0059] See also Figure 4 As shown in Figure 2, facial regions of the same identity show self-similarity without considering domain differences. Therefore, we use spatial self-similarity to measure the difference between x and ^y.
[0060] Optionally, the calculation formula for the structured loss of the i-th point is as follows:
[0061]
[0062] in, For and The cosine similarity of For x i The vector of for is a vector of , ζ is the scale factor.
[0063] Furthermore, the acquired images are from the CASIA NIR-VIS 2.0 upper and middle school college student dataset.
[0064] See also Figure 3 As shown in Figure 2, in order to achieve diversity and identity consistency, a framework CoConGAN for heterogeneous face conversion is proposed. The input image x is combined with a query hidden code z and a positive sample z + , and N negative samples z sampled from a Gaussian distribution - , input into the generator G, and get the same output as and The output is then fed into the encoder G enc Get the output encoding embedding e, e + and The encoder shares some layers with G. Latent space contrast regularization is used to encourage diversity. The pre-trained model F is used to extract the image x, The perceptual characteristics of x , The spatial correlation map S of the features x , Used to ensure identity consistency.
[0065] In summary, the present invention extends HFT to a more challenging and novel few-shot unsupervised heterogeneous face-to-face translation environment. In addition, instead of using various conventional terms for complex restrictions, only three terms are explicitly specified to jointly guide HFT.
[0066] The specific contents of the evaluation of CoConGAN through qualitative and quantitative experiments are as follows:
[0067] Datasets: CASIA NIRVIS 2.0 and CUHK student. For CASIA NIR-VIS 2.0, to ensure that the identities in NIR and VIS are not repeated, we select the images of the first 290 people from NIR and the images of the last 290 people from VIS. For CUHK student, we adjust the order of images in the datasets photo and sketch to ensure that images in the two datasets are not paired.
[0068] Evaluation indicators: KID, IS and LPIPS are used for evaluation. KID and IS are used to evaluate the visual quality of generated images, and LPIPS is used to evaluate the diversity of generated images.
[0069] Comparison method: The proposed method is compared with DivCo and DVGFace. DivCo uses the same unpaired few-shot training set as CoConGAN, while DVG Face only uses paired training sets with the same identity in the source and target domains. During testing, we only provide the target dataset for DivCo, DVG Face, and CoConGAN.
[0070] Qualitative assessment: Figure 5 shows a visual comparison of HFT results. We can see that DVG faces rely heavily on identity matching consistency during training. The visual effect of images generated by DivCo is improved, but due to the lack of identity consistency, there is still an identity difference between the trained image y and the input image x (i.e., not the same person). CoConGAN with latent contrast and structural contrast regularization not only generates diverse samples, but also maintains identity consistency.
[0071] Quantitative evaluation: As shown in Table 1, the quantitative comparison with KID, IS, and LPIPS, the arrow indicates the better direction. Specifically, there is a trade-off between diversity and identity consistency. Therefore, under the premise of identity consistency, the sample diversity of CoConGAN will inevitably be affected, and since DVG-Face is not restricted by identity consistency, the value of LPIPS is the largest.
[0072] Table 1.
[0073]
[0074] Conclusion: Heterogeneous face images appear in many forms: high resolution and low resolution; sketches and line drawings; facial expressions and facial animations; thermal infrared, near infrared and visible light face images. This study focuses on the transformation of heterogeneous faces. To this end, we propose a CoConGAN and conduct multiple experiments on two heterogeneous face datasets to verify the effectiveness of the method from qualitative and quantitative perspectives.
[0075] The above description of the disclosed embodiments is provided in a progressive manner to enable those skilled in the art to implement or use the present invention. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention will not be limited to the embodiments shown herein, but rather to the widest scope consistent with the principles and novel features disclosed herein.
Claims
1. A small sample heterogeneous face conversion method based on collaborative contrast loss, characterized in that: The following steps are involved: Obtain an image, a hidden factor, a positive sample and N negative samples, and use the obtained image as the source image; Input the source image, an invisible factor, a positive sample and N negative samples into the trained generator, and output the first image group and the second image group with the same identity; The perceptual features of the source image and the first image group are extracted respectively by using the perceptual feature extraction model, and feature space correlation graphs of the source image and the first image group are obtained, and used as structural contrast constraints; Encode the first image group and the second image group using an encoder, and embed the obtained code into features of an image, features of a positive sample image, and features of N negative sample images as latent space contrast constraints; The final transformed face image is obtained by using structural contrast constraints and latent space contrast constraints; The training process of the generator is as follows: input the source image and the hidden factor, and output the trained image, where the hidden factor follows a normal distribution; The total loss of the generator is calculated as follows: Among them, L adv (G, D) is the adversarial loss function, L Lacontra (G) is the potential contrast regularization function, L Stcontra (G) is the structural contrast regularization function; γ is a hyperparameter, G is the generator, and D is the discriminator; The adversarial loss calculation formula of the generator is as follows: L adv (G,D)=E[log(D(y))]+E[log(1-D(x,z))] (2) Among them, x is the input image from the source domain, y is the reference image from the target domain, and z is the invisible factor; The cooperative contrastive loss of the generator includes latent contrastive regularization loss and structural contrastive regularization loss; The latent contrast regularization loss is calculated as follows: Among them, τ is the scale used to control the size of the loss, e is the feature of the image, and e + is the feature of the positive sample image; The formula for calculating the structural contrast regularization loss is as follows: in, is the structural loss of the i-th point, N st is the number of points; The calculation formula of the structured loss of the i-th point is as follows: in, For and The cosine similarity of For x i The vector of for is a vector of , ζ is the scale factor.
2. According to claim 1, a small sample heterogeneous face conversion method based on collaborative contrast loss is characterized in that: The acquired images are from the CASIA NIR-VIS2.0 upper and middle school student dataset.
