Heterogeneous face recognition method based on multi-feature fusion reconstruction generation

Abstract

The invention relates to a heterogeneous face recognition method, in particular to a heterogeneous face recognition method based on multi-feature fusion reconstruction generation. The method comprises the following steps: acquiring a target face image, and inputting the target face image into a trained generator to obtain a converted face image corresponding to the target face image; and inputting the converted face image corresponding to the target face image into the trained classifier to obtain an identity recognition result of the target face image. According to the invention, a cyclic generative adversarial network model is formed by a generator based on an RSTB deep identity feature extractor and a discriminator, and a real face image with a reasonable structure and maintained identity can be generated in an unsupervised manner through effective fusion of identity features, style features, content features, local features and global features. When the method is used for heterogeneous face recognition, the face recognition performance can be effectively improved; the corresponding identity feature loss and the style feature loss accelerate the convergence speed of the model and improve the generalization ability of the model.

Description

Technical Field

[0001] The present invention relates to a heterogeneous face recognition method, and in particular to a heterogeneous face recognition method based on multi-feature fusion reconstruction generation. Background Art

[0002] Heterogeneous face recognition (HFR) refers to determining the identity of a face image from given face images of different modalities (face images collected by different types of imaging devices). Typical heterogeneous face recognition includes near-infrared-visible light (NIR-VIS) face recognition and sketch-visible light (Sketch-VIS) face recognition. Its goal is to expand the applicability of face recognition systems in challenging scenarios, enable cross-domain recognition of face images, and have broad application prospects and economic value in security monitoring and law enforcement.

[0003] At present, the main challenge facing heterogeneous face recognition is the difference between different modalities. In order to deal with this problem, heterogeneous face generation or modality-invariant feature extraction methods are usually used to reduce cross-domain differences and thus improve recognition accuracy. Modality-invariant feature extraction methods extract modality-independent feature representations from face images by designing or learning effective face feature extraction methods, thereby reducing modality differences when converting heterogeneous face images into feature representations. Most of the current mainstream modality-invariant feature extraction methods are based on deep learning. For example, the literature Du, H. Shi, Y. Liu, D. Zeng and T. Mei, "Towards NIR-VIS Masked Face Recognition," in IEEE SignalProcessing Letters, vol. 28, pp. 768-772, 2021. proposed a method for NIR-VIS occluded face recognition, which eliminates occlusion errors and enhances cross-modal face recognition performance and robustness by learning modality-invariant features and utilizing feature consistency under occlusion. The literature A. George and S. Marcel, "Bridging the Gap: Heterogeneous Face Recognition with Conditional Adaptive Instance Modulation," 2023 IEEE International Joint Conference on Biometrics (IJCB), Ljubljana, Slovenia, 2023, pp. 1-10. proposed a conditional adaptive instance modulation (CAIM) method, which conditionally modulates the intermediate feature map of the pre-trained face recognition network to reduce the domain difference between visible light and other modalities and enhance the discriminative performance of face representation. However, these methods still face some challenges. For example, when dealing with NIR-VIS face recognition, it is more difficult to complicate feature extraction and maintain modality consistency; and the method based on conditional adaptive instance modulation may lead to overfitting on small-scale cross-modal datasets. In addition, traditional deep convolutional neural networks also have difficulty extracting modality-independent features when dealing with large cross-modal differences.

[0004] Compared with the modality-invariant feature extraction method, the heterogeneous face generation method converts the face image from one modality to another by using conditional or unconditional image synthesis technology to match the face with the same modality. The heterogeneous face generation method has more direct image comparison ability, high-precision feature alignment, utilization of existing powerful models, enhanced data consistency and high flexibility, thus significantly improving the overall recognition effect and practicality. Early literature Q.Liu, X.Tang, H.Jin, H.Lu, and S.Ma, "A nonlinear approach for face sketch synthesis and recognition," in Proc.IEEE Comput.Soc.Conf.Comput.Vis.Pattern Recognit., SanDiego, CA, USA, Jun.2005, pp.1005–1010. proposed a small block independent synthesis algorithm by dividing the face image into overlapping small blocks and reconstructing them by linear combination in the training set, but because the compatibility relationship between adjacent small blocks is ignored, the overall consistency of the generated face image is poor. Reference Fang, Y., Deng, W., Du, J., & Hu, J. (2020). Identity-aware CycleGAN for face photo-sketch synthesis and recognition. Pattern Recognit., 102, 107249. In the reference J. Zhu, T. Park, P. Isola, and AAE fros, "Unpaired Image-to-Image Translation Using Cycle-Consistent Adversarial Networks," in 2017 IEEE International Conference on Computer Vision (ICCV), 2017, pp. 2242-2251., the IACycleGAN model with identity-aware supervision is introduced into the CycleGAN framework. By combining the bidirectional conversion generator and discriminator with perceptual loss, the synthesis quality is improved and the facial features for recognition are retained.The literature C.Fu, X.Wu, Y.Hu, H.Huang and R.He, "DVG-Face: Dual Variational Generation for Heterogeneous Face Recognition," in IEEE Transactions on Pattern Analysis and Machine Intelligence, 2022, pp.2938-2952. proposed the DVG-Face model, which introduces a dual generation mechanism in the variational autoencoder (VAE) framework to learn the joint distribution of paired heterogeneous images to improve the synthesis quality and retain facial features for recognition. The above literatures all use two generators to perform bidirectional conversion of heterogeneous images, and use identity preservation loss and contrastive learning mechanisms to ensure high-fidelity identity information, thereby enhancing recognition performance. Although identity-aware supervision and identity-aware loss are introduced in these models respectively, there is still a problem of insufficient fusion of local features and global features, and they all need to use a large amount of high-quality paired data for training. In the process of facial modality conversion, the failure to properly handle local details and global consistency will reduce the quality of the generated image, making it lack of realism and naturalness, and causing obvious visual flaws and artifacts, thus affecting the face recognition effect in practical applications. Summary of the invention

[0005] In view of the problems of insufficient fusion of local details and global features in the above recognition methods based on heterogeneous face generation, the present invention provides a heterogeneous face recognition method based on multi-feature fusion reconstruction generation. This method constructs a recurrent generative adversarial network composed of a generator that integrates RSTB (Residual Swin Transformer Block) deep identity feature extractor and a discriminator based on RSTB, and reconstructs the deep identity features and content features of hand-drawn faces with the style features of visible light faces to achieve modal conversion from hand-drawn faces to visible light faces. Heterogeneous face recognition based on the converted face images is conducive to improving recognition performance.

[0006] First, the present invention proposes a generator that integrates the RSTB deep identity feature extractor. RSTB combines the window block attention mechanism and residual connection, which can effectively perform parallel computing and accelerate the training and reasoning process of the model. The feature extractor has a hierarchical windowed attention mechanism, which can extract features at different scales and resolutions, and is conducive to the fusion of local features and global features, so that the generated face picture retains local details and global consistency at the same time. At the same time, the RSTB deep feature extractor can effectively process long-distance dependencies, help the generator understand the image structure globally, and generate realistic and reasonable real face images. Secondly, the present invention introduces a new identity feature loss function and style feature loss function. The identity feature loss function directly calculates the difference between the generated visible light face and the hand-painted face image to measure the retention of the identity feature and accelerate the convergence speed of the identity feature. The style feature loss function adds skewness calculation when calculating the difference between the generated image and the visible light style face image to prevent the model from excessively biasing its style towards the training data set when generating faces, thereby improving the generalization ability of the model to generate samples. Finally, the present invention constructs a discriminator model based on RSTB. RSTB can not only effectively extract local details of the image, but also ensure the consistency of the overall effect, and fully integrate global features and local features. Using RSTB for feature extraction and discrimination enhances the diversity of contrast features, which is conducive to improving the performance of the discriminator. The introduction of RSTB increases the complexity of the discriminator model, which is balanced with the complexity of the generator model, thereby enhancing the stability of training.

[0007] In order to achieve the above object, the present invention provides the following technical solutions:

[0008] A heterogeneous face recognition method based on multi-feature fusion reconstruction generation, the method comprising: obtaining a target face image, inputting the target face image into a trained generator, and obtaining a converted face image corresponding to the target face image; inputting the target face image and its corresponding converted face image into a trained classifier, and obtaining an identity recognition result corresponding to the target face image; wherein the training process comprises:

[0009] Step 1, obtaining a pair of real face images and their corresponding identity tags, wherein the pair of real face images is a hand-drawn face image and a visible light face image of the same object;

[0010] Step 2: Input the real face image pair into the generator, obtain content features or style features through the VGG feature extractor, obtain shallow identity features through the shallow feature extractor, and obtain deep identity features by processing the shallow identity features through the deep feature extractor; fuse the style features of the hand-drawn face image of the same object with the deep identity features and content features of the visible light face image, and obtain a reconstructed hand-drawn face image through the face reconstructor; fuse the style features of the visible light face image of the same object with the deep identity features and content features of the hand-drawn face image, and obtain a reconstructed visible light face image through the face reconstructor;

[0011] Step 3, inputting the real visible light face image and the reconstructed visible light face image into the discriminator, and obtaining the discrimination result of whether the input visible light face image is the reconstructed visible light face image through the discriminator; inputting the real hand-drawn face image and the reconstructed hand-drawn face image into the discriminator, and obtaining the discrimination result of whether the input hand-drawn face image is the reconstructed hand-drawn face image through the discriminator;

[0012] Step 4, inputting the reconstructed visible light face image and / or the reconstructed hand-drawn face image into a classifier, and obtaining an identity prediction result of the input face image through the classifier;

[0013] Step 5, calculate the style loss value, content loss value and identity loss value of step 2, the adversarial loss value and cycle consistency loss value of the discrimination result of step 3 and the real face image, and the classification loss value of the identity prediction result of step 4 and the real identity label; return the loss value to update the training parameters until the training is completed.

[0014] Compared with the prior art, the method of the present invention has the following advantages and beneficial effects:

[0015] 1) The recurrent generative adversarial network model proposed in the present invention, which is composed of a generator based on RSTB deep identity feature extractor and a discriminator based on RSTB deep identity feature extractor, can unsupervisedly generate real face images with reasonable structure and identity preservation through the effective fusion of identity features, style features, content features, local features and global features. Using it for heterogeneous face recognition can effectively improve the performance of face recognition; 2) The identity feature loss and style feature loss introduced in the present invention accelerate the convergence speed of the model while further improving the generalization ability of the model. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] In order to make the purpose, technical solutions and advantages of the present invention more clear, the present invention will be further described in detail below with reference to the accompanying drawings, in which:

[0017] Figure 1 This is a flow chart of a heterogeneous face recognition method in an embodiment of the present invention;

[0018] Figure 2 This is a flow chart of heterogeneous face recognition model training in an embodiment of the present invention;

[0019] Figure 3 This is a schematic diagram of heterogeneous face recognition training in an embodiment of the present invention;

[0020] Figure 4 This is a schematic diagram of the RSTB network structure in an embodiment of the present invention;

[0021] Figure 5 This is a schematic diagram of the STL network structure in an embodiment of the present invention;

[0022] Figure 6 This is a schematic diagram of the generator network structure in an embodiment of the present invention;

[0023] Figure 7 This is a schematic diagram of a heterogeneous face recognition model test in an embodiment of the present invention. DETAILED DESCRIPTION

[0024] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0025] like Figure 1 As shown, the present embodiment provides a method for heterogeneous face recognition by combining feature decoupling and modal conversion, the method comprising: obtaining a target face image, inputting the target face image into a trained generator, and obtaining a converted face image corresponding to the target face image; inputting the target face image and its corresponding converted face image into a trained classifier, and obtaining an identity recognition result corresponding to the target face image;

[0026] In some embodiments of the present invention, the heterogeneous face recognition method of this embodiment is mainly used for identifying the types of heterogeneous faces, such as the recognition of face images including real faces, cartoon faces and hand-drawn faces. The target face image can be an image captured by a visible light camera under the premise of user authorization, as well as a hand-drawn image called from a hand-drawn database. In practical applications, live video in a specific scene is collected by a camera or a cluster monitoring device, and the head in the live video is identified and marked using image extraction software. The head is cut out from the video screen based on the mark to generate a target face image. The target face image is passed through a trained generator and classifier to obtain the identity recognition result corresponding to the target face image.

[0027] like Figure 2 As shown in Figure 1, the training process corresponding to the heterogeneous face recognition model includes:

[0028] Step 1, obtaining a pair of real face images and their corresponding identity tags, wherein the pair of real face images is a hand-drawn face image and a visible light face image of the same object;

[0029] In the embodiment of the present invention, the images of the training set are derived from the CUFSF (CUHK Face Sketch FERET Database) data set, including visible light face images of 606 people, each visible light face image has a matching hand-drawn face image, and these visible light face images and matching hand-drawn face images have corresponding identity tags, which can distinguish the face images of different objects. Resize all images in the training set to 128×128×3.

[0030] For example, at the beginning of training, it is necessary to select N pairs of heterogeneous visible light face images and hand-drawn face images from the training set, that is, to select 2N face images, or N pairs of face images can be selected. The N pairs of face images can include heterogeneous face image pairs, and can also include face image pairs with different image attributes of the two face images, that is, the number of visible light face images and hand-drawn face images does not completely correspond; this embodiment is explained by selecting N pairs of heterogeneous face images, each pair of heterogeneous faces in the N pairs of heterogeneous face images corresponds to a target face in N different face objects, the face image with serial number a and the face image with serial number N+a correspond to the same face object, the face image with serial number a can be a visible light face image, and the face image with serial number N+a is a hand-drawn face image, and both have the same identity label a', which is used to identify the object identity of the face image.

[0031] Step 2: Input the real face image pair into the generator, obtain content features or style features through the VGG feature extractor, obtain shallow identity features through the shallow feature extractor, and obtain deep identity features by processing the shallow identity features through the deep feature extractor; fuse the style features of the hand-drawn face image of the same object with the deep identity features and content features of the visible light face image, and obtain a reconstructed hand-drawn face image through the face reconstructor; fuse the style features of the visible light face image of the same object with the deep identity features and content features of the hand-drawn face image, and obtain a reconstructed visible light face image through the face reconstructor;

[0032] In the embodiment of the present invention, in the training stage, for the convenience of description, as Figure 3As shown, the case of converting hand-drawn face images to visible light face images is used as an example to illustrate the conversion of the i-th visible light face image in the training set. and hand-drawn face images At the same time, it is input into the generator G. In the generator, The content features f of the hand-drawn face image are obtained by the feature extractor VGG and the deep identity feature extractor respectively. c and deep identity features id . The style feature f of the visible light face image is obtained through the feature extractor VGG s . The hand-drawn face image set I C The i-th hand-drawn face image in The extracted content features f c , deep identity features id and the i-th visible light face image I S Medium style features s After fusion, it is input into the face reconstructor to obtain the i-th hand-drawn face image after fusion and reconstruction. Corresponding visible light face image Similarly, the i-th visible light face image after fusion and reconstruction is obtained Corresponding hand-drawn face image

[0033] In the embodiment of the present invention, the RSTB added to the generator G has a hierarchical windowed attention mechanism, which extracts features at different scales and resolutions, which is conducive to the fusion of local features and global features, so that the generated face image maintains both details and global consistency. This makes the deep feature extraction of the input cross-modal face more detailed. RSTB is composed of STL (Swin Transformer Layer). The network structure of RSTB and STL is as follows: Figure 4 and Figure 5 As shown in Figure 2. The feature extractor VGG in the generator G uses a custom VGG19 network structure. Figure 6 The generator network structure shown.

[0034] Step 3: Input the real visible light face image and the reconstructed visible light face image into the discriminator, and obtain the discriminant result of whether the input visible light face image is the reconstructed visible light face image through the discriminator; input the real hand-painted face image and the reconstructed hand-painted face image into the discriminator, and obtain the discriminant result of whether the input hand-painted face image is the reconstructed hand-painted face image through the discriminator; in this embodiment of the present invention, the hand-painted face image with the i-th hand-painted face image is input into the discriminator, and obtain the discriminant result of whether the input hand-painted face image is the reconstructed hand-painted face image through the discriminator. Corresponding visible light face image And the i-th hand-drawn face image generated by generator G Corresponding visible light face image The input is fed into the discriminator D to determine whether these input visible light face images are reconstructed and synthesized face images; similarly, it can determine whether the input hand-drawn face images are reconstructed and synthesized face images.

[0035] Among them, the discriminator D uses a gradually growing convolutional neural network, which makes the model more stable when training high-resolution images by gradually increasing the resolution. The RSTB module is added to the discriminator D for deep feature extraction, which improves the discriminator's ability to discriminate generated images. At the same time, it also balances the complexity of the discriminator model and the generator model, thereby enhancing the stability of training. Discrete wavelet transform is also introduced in the discriminator D to check the spectral differences between face images, which to a certain extent suppresses the phenomenon of artifacts in the generated visible light face images.

[0036] It should be noted that, since the present invention can convert visible light face images into hand-painted face images, and can also convert hand-painted face images into visible light face images, the model can include two generators G X and G Y , two discriminators D X and D Y ; Generator G X The visible light face image can be processed, and the generator G Y Hand-drawn face images can be processed, and the discriminator D X The real visible light face image and the reconstructed visible light face image can be processed. The discriminator D Y Real hand-drawn face images and reconstructed hand-drawn face images can be processed.

[0037] Step 4, inputting the reconstructed visible light face image and / or the reconstructed hand-drawn face image into a classifier, and obtaining an identity prediction result of the input face image through the classifier;

[0038] In the embodiment of the present invention, the real visible light face image and reconstructed visible light face image Input to classifier C, get the facial features of the input face image, and predict the identity of the input face image; similarly, the real hand-drawn face image and reconstructed hand-drawn face images Input to classifier C, obtain the facial features of the input face image, and predict the identity of the input face image.

[0039] Among them, in some embodiments, the classifier adopts the LightCNN network structure pre-trained on the MS-Cleleb-1M dataset. LightCNN (Light Convolutional Neural Networks) is a lightweight convolutional neural network, which is particularly suitable for tasks such as face recognition. The design of its network structure aims to reduce model parameters and computational complexity while maintaining high recognition performance; in other possible implementation methods, the classifier can also adopt existing conventional classification models, such as Bayesian classifiers, neural network classifiers, etc., and the present invention does not specifically limit this.

[0040] Step 5, calculate the style loss value, content loss value and identity loss value of step 2, the adversarial loss value and cycle consistency loss value of the discrimination result of step 3 and the real face image, and the classification loss value of the identity prediction result of step 4 and the real identity label; return the loss value to update the training parameters until the training is completed. In the embodiment of the present invention, considering that the loss function is in addition to the adversarial loss function commonly used in the same type of method, the present invention further includes an identity loss function, a cycle consistency loss function, a style feature loss function, a content feature loss function, and a classification loss function;

[0041] First, the style feature loss function and the content feature loss function are used to constrain the feature extraction process of the image, respectively, so that the feature extractor only extracts the content features of the hand-drawn face image to keep the content information and face feature information before the conversion, and the style feature extractor only extracts the style feature information of the visible light face image, so that the converted face image retains its content information while having the style information of the visible light face. The content feature loss function, style feature loss and feature extraction overall loss function formulas are defined as follows:

[0042]

[0043] L total =αL c +βL s (4)

[0044] Among them, w j represents the weight of the j-th layer feature map, represents the j-th layer feature map of the i-th reconstructed face image extracted by the VGG feature extractor, represents the j-th layer feature map of the ith real face image extracted by the VGG feature extractor. If the reconstructed face image is a visible light face image, the real face image corresponds to a hand-drawn face image. If the reconstructed face image is a hand-drawn face image, the real face image corresponds to a visible light face image. For the convenience of representation, the above formula is used as an example to illustrate the loss function corresponding to the case of converting a hand-drawn face image to a visible light face image. μ(·) represents the mean, σ(·) represents the variance, and γ(·) represents the skewness. Formula (3) is the skewness formula, and ∥·∥2 represents the L2 norm.

[0045] From formula (1), we can see that the content loss function reflects the identity information difference between the output image and the input image by directly calculating the difference between the feature maps of each layer. From formula (2), we can see that the style loss function calculates the output image With the input style image The L2 norm and skewness of the mean and variance of the image are used to evaluate the style difference between images. The complementary measurement of style difference by skewness is conducive to making the style of the generated image closer to the style image, while avoiding overfitting of the model. Formula (4) is the weighted sum of style loss and identity loss, where μ and β are the loss coefficients of identity loss and style loss respectively, which are used to adjust the feature proportion. To achieve the effect of adjusting the bias of the image, α=100000 and β=0.01 are set according to the experiment.

[0046] In some preferred embodiments of the present invention, a standard deviation weight w is introduced for each layer of feature maps. j , weight w j The introduction of makes the high-variation area occupy a larger weight in the loss, which can better preserve the details in the face image. The weight w j It is the inverse of the standard deviation of the feature map of the face image, which is used to weight the difference. Feature maps with large standard deviations (high-variation areas) have a larger weight in the loss. Feature maps with large standard deviations represent areas with more variations, which usually contain more details and information. By applying greater weights to these areas, the loss function can better focus on these important areas, thereby better preserving these details when generating images. Therefore, the calculation formula for the weight of the j-th layer feature map is expressed as:

[0047]

[0048] Among them, φ j (I c ) represents the j-th layer feature map of all real face images extracted by the VGG feature extractor; σ represents the standard deviation.

[0049] In some preferred embodiments of the present invention, based on the above embodiments, a more refined control is performed on the feature map, using a pixel-level weight W j (p), can be expressed as:

[0050] W j (p) = 1-0.1·d M (L,P) (6)

[0051] Among them, d M (L,P) represents the Manhattan distance between the pth pixel and the center pixel L. Pixel-level weights allow precise control over the importance of each pixel during the generation process. This approach ensures that pixels in important feature areas (such as facial features) are more important, thereby enhancing the accuracy and details of the generated image in these key areas. Therefore, the updated content loss value is expressed as:

[0052]

[0053] Among them, W j (p) represents the weight of the p-th pixel in the j-th layer feature map, represents the feature value of the p-th pixel in the j-th layer feature map of the i-th reconstructed face image extracted by the VGG feature extractor, It represents the eigenvalue of the p-th pixel in the j-th feature map of the ith real face image extracted by the VGG feature extractor; if the reconstructed face image is a visible light face image, the corresponding real face image is a hand-drawn face image; if the reconstructed face image is a hand-drawn face image, the corresponding real face image is a visible light face image.

[0054] The cycle consistency loss function is a key component in the network, which ensures that the image generated by the conversion from one domain to another domain and then back to the original domain is as close to the original image as possible. The identity loss function is used in the network to encourage the generator to generate images consistent with the input image when the input image is already from the target domain. The formulas of the cycle consistency loss function and the identity loss function are defined as follows:

[0055]

[0056] where E[·] is the expected value, φ i(·) represents the feature map of the i-th layer, ∥·∥1 represents the L1 norm, and G(·) represents the generator. Formula (8) is the identity loss function, formulas (9) and (10) are the cycle consistency loss functions, and formula (11) is the sum of the cycle consistency loss function values. The identity loss value between the output image and the input hand-drawn face image is obtained by calculating the L1 norm between the output image and the input hand-drawn face image for parameter update. The generated result image is maintained by calculating the difference between the output result generated by the generator G and the hand-drawn face image and the visible light face image. identity information and style information;

[0057] Next is the classification loss function, which is constructed using the softmax cross entropy loss function, where the input of the classifier is the visible light image I generated by the generator o , the classification loss updates the classifier and optimizes the generator at the same time, and the formula is as follows:

[0058]

[0059] Among them, y ij represents the true label value of sample i in category j, is the predicted probability of sample i in category j.

[0060] The adversarial loss uses the adversarial loss function of the CycleGAN network, and the formula is:

[0061]

[0062] in, Represents the i-th hand-drawn face image The corresponding visible light face image is generated. Indicates the generation of visible light face images Predict the probability of whether it is a synthetic image, is the true label.

[0063] The total loss function is:

[0064] L=λ id L id +λ cycle L cycle +λ total L total +λ adv L adv +λ clf L clf (14)

[0065] Among them, λ id , cycle , total , adv ,clf Represents the weight of the corresponding loss importance. Through a lot of experimental experience, id , cycle , total , adv , clf Set to 5, 10, 5, 1, and 0.1 respectively.

[0066] Alternating training of each module in the network can optimize and improve the modality conversion face generation task, feature extraction task and heterogeneous face recognition task. The extraction of style features and identity features enables the modality conversion face generation task to be completed smoothly, so that the generated face image has both the identity information of the hand-drawn face image and the style information of the visible light face. The generated face image is supplemented with data and the extracted identity features are integrated into the heterogeneous face recognition classifier, which can promote the improvement of heterogeneous face recognition performance.

[0067] In some preferred embodiments, the present invention further adds a testing phase to the above training process, such as Figure 7 As shown, in the testing phase, there are two methods: qualitative testing and quantitative testing to verify the effect of the model.

[0068] In the qualitative test, the hand-drawn face image to be converted and the visible light face image are input into the generator G to obtain the face image after modal conversion. The effect of generating visible light faces can be verified by directly observing the quality of the synthesized modal conversion face image.

[0069] In the quantitative test, the face image generated by the generator G is The face dataset to be detected with labels is input into the trained classifier C respectively, and the Rank recognition rate is calculated by calculating the difference between the feature maps, that is, the face image is generated. The probability of successfully searching for the correct gallery image for the first time. The higher the Rank recognition rate, the better the model effect.

[0070] Although embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present invention, and that the scope of the present invention is defined by the appended claims and their equivalents.

Claims

1. A heterogeneous face recognition method based on multi-feature fusion reconstruction generation, characterized in that: The method comprises: obtaining a target face image, inputting the target face image into a trained generator, and obtaining a converted face image corresponding to the target face image; inputting the target face image and its corresponding converted face image into a trained classifier, and obtaining an identity recognition result corresponding to the target face image; wherein the training process comprises: Step 1, obtaining a pair of real face images and their corresponding identity tags, wherein the pair of real face images is a hand-drawn face image and a visible light face image of the same object; Step 2: Input the real face image pair into the generator, obtain content features or style features through the VGG feature extractor, obtain shallow identity features through the shallow feature extractor, and obtain deep identity features by processing the shallow identity features through the deep feature extractor; fuse the style features of the hand-drawn face image of the same object with the deep identity features and content features of the visible light face image, and obtain a reconstructed hand-drawn face image through the face reconstructor; fuse the style features of the visible light face image of the same object with the deep identity features and content features of the hand-drawn face image, and obtain a reconstructed visible light face image through the face reconstructor; Step 3, inputting the real visible light face image and the reconstructed visible light face image into the discriminator, and obtaining the discrimination result of whether the input visible light face image is the reconstructed visible light face image through the discriminator; inputting the real hand-drawn face image and the reconstructed hand-drawn face image into the discriminator, and obtaining the discrimination result of whether the input hand-drawn face image is the reconstructed hand-drawn face image through the discriminator; Step 4, inputting the reconstructed visible light face image and / or the reconstructed hand-drawn face image into a classifier, and obtaining an identity prediction result of the input face image through the classifier; Step 5, calculate the style loss value, content loss value and identity loss value of step 2, the adversarial loss value and cycle consistency loss value of the discrimination result of step 3 and the real face image, and the classification loss value of the identity prediction result of step 4 and the real identity label; return the loss value to update the training parameters until the training is completed.

2. According to claim 1, a heterogeneous face recognition method based on multi-feature fusion reconstruction generation is characterized in that: The discriminator includes a deep feature extractor, a discrete wavelet transform layer HarrTransform, an RGB image processing layer, a convolution layer and a linear layer.

3. A heterogeneous face recognition method based on multi-feature fusion reconstruction generation according to claim 1 or 2, characterized in that: The deep feature extractor includes multiple STL layers and an LN normalization layer; each STL layer is connected in sequence, and the output end of the last STL layer is connected to the input end of the LN normalization layer; the input end of the first STL layer is superimposed with the output end of the LN normalization layer as the output end of the next layer.

4. According to claim 1, a heterogeneous face recognition method based on multi-feature fusion reconstruction generation is characterized in that: The calculation formula of the content loss value is expressed as: Among them, w j represents the weight of the j-th layer feature map, represents the j-th layer feature map of the i-th reconstructed face image extracted by the VGG feature extractor, Represents the j-th layer feature map of the ith real face image extracted by the VGG feature extractor; if the reconstructed face image is a visible light face image, then the corresponding real face image is a hand-drawn face image; if the reconstructed face image is a hand-drawn face image, then the corresponding real face image is a visible light face image.

5. According to claim 4, a heterogeneous face recognition method based on multi-feature fusion reconstruction generation is characterized in that: The calculation formula of the weight of the j-th layer feature map is expressed as: Among them, φ j (I c ) represents the j-th layer feature map of all real face images extracted by the VGG feature extractor; σ represents the standard deviation.

6. The method for heterogeneous face recognition based on multi-feature fusion reconstruction generation according to claim 1 is characterized in that: The calculation formula of the content loss value is expressed as: Among them, W j (p) represents the weight of the p-th pixel in the j-th layer feature map, represents the feature value of the p-th pixel in the j-th layer feature map of the i-th reconstructed face image extracted by the VGG feature extractor, It represents the eigenvalue of the p-th pixel in the j-th feature map of the ith real face image extracted by the VGG feature extractor; if the reconstructed face image is a visible light face image, the corresponding real face image is a hand-drawn face image; if the reconstructed face image is a hand-drawn face image, the corresponding real face image is a visible light face image.

7. The method for heterogeneous face recognition based on multi-feature fusion reconstruction generation according to claim 6 is characterized in that: The calculation formula of the content loss value is expressed as: W j (p)=1-0.1·d M (L,P) Among them, d M (L, P) represents the Manhattan distance between the p-th pixel and the center pixel L.

8. The method for heterogeneous face recognition based on multi-feature fusion reconstruction generation according to claim 1 is characterized in that: The calculation formula of the identity loss value is expressed as: Among them, E[·] is the expected value, represents the feature map of the jth layer of the i-th reconstructed face image extracted by the deep feature extractor, It represents the feature map of the jth layer of the ith real face image extracted by the deep feature extractor. If the reconstructed face image is a visible light face image, then the real face image corresponds to the visible light face image. If the reconstructed face image is a hand-drawn face image, then the real face image corresponds to the hand-drawn face image.

9. The method for heterogeneous face recognition based on multi-feature fusion reconstruction generation according to claim 1, characterized in that: The calculation formula of the style loss value is expressed as: Among them, N l represents the number of feature layers, μ(·) represents the mean, σ(·) represents the variance, and γ(·) represents the skewness. represents the feature map of the jth layer of the i-th reconstructed face image extracted by the VGG feature extractor, represents the feature map of the jth layer of the ith real face image extracted by the VGG feature extractor, ||·||2 represents the L2 norm, if the reconstructed face image is a visible light face image, then the real face image corresponds to the visible light face image, if the reconstructed face image is a hand-painted face image, then the real face image corresponds to the hand-painted face image.

10. The method for heterogeneous face recognition based on multi-feature fusion reconstruction generation according to claim 9, characterized in that: The calculation formula of the skewness is expressed as: Among them, φ j (I) represents the j-th layer feature map of all real face images extracted by the VGG feature extractor.

Images