A heterogeneous face recognition method based on multi-feature fusion reconstruction generation

Abstract

The present application relates to a kind of heterogeneous face recognition methods, in particular to a kind of heterogeneous face recognition method based on multi-feature fusion reconstruction generation;The method includes obtaining target face image, input target face image into the generator after training, obtain the conversion face image corresponding to target face image;The conversion face image corresponding to target face image is input into the classifier after training, and the identity recognition result of target face image is obtained.The generator and discriminator based on RSTB deep identity feature extractor of the present application are constituted into cyclic generative adversarial network model, can be generated by the effective fusion of identity feature, style feature, content feature and local feature and global feature Unsupervisedly generate real face image with reasonable structure, identity preservation, it can effectively improve the performance of face recognition when being used for heterogeneous face recognition;Corresponding identity feature loss and style feature loss accelerate the convergence speed of model, while also improve the generalization ability of model.

Description

TECHNICAL FIELD

[0001] The present application relates to a heterogeneous face recognition method, in particular to a heterogeneous face recognition method based on multi-feature fusion reconstruction generation. BACKGROUND

[0002] Heterogeneous face recognition (HFR) refers to determining the identity of a face image from given face images of different modalities (face images collected by different types of imaging devices). Typical heterogeneous face recognition includes near-infrared-visible light (NIR-VIS) face recognition and hand-drawn face-visible light (Sketch-VIS) face recognition, etc. The goal is to expand the applicability of face recognition systems in challenging scenarios, enabling cross-domain recognition of face images, and having wide application prospects and economic value in the fields of security monitoring and public security law enforcement, etc.

[0003] Currently, the main challenge for heterogeneous face recognition is the difference between different modalities. To address this issue, heterogeneous face generation or modality-invariant feature extraction methods are usually used to reduce cross-domain differences and improve recognition accuracy. Modality-invariant feature extraction methods extract features that are independent of modalities by designing or learning effective face feature extraction methods, thereby reducing the modal difference when converting heterogeneous face images into feature representations. Current mainstream modality-invariant feature extraction methods are mostly based on deep learning. For example, Du, H. Shi, Y. Liu, D. Zeng and T. Mei, "Towards NIR-VIS Masked Face Recognition," in IEEE Signal Processing Letters, vol. 28, pp. 768-772, 2021. proposed a method for NIR-VIS occluded face recognition, which learns modality-invariant features, uses feature consistency under occlusion, eliminates occlusion errors, and enhances the performance and robustness of cross-modal face recognition. A. George and S. Marcel, "Bridging the Gap: Heterogeneous Face Recognition with Conditional Adaptive Instance Modulation," 2023 IEEE International Joint Conference on Biometrics (IJCB), Ljubljana, Slovenia, 2023, pp. 1-10. proposed a conditional adaptive instance modulation (CAIM) method, which modulates the intermediate feature maps of a pre-trained face recognition network to reduce the domain difference between visible light and other modalities and enhance the discriminative performance of face representation. However, these methods still face some challenges. For example, when dealing with NIR-VIS face recognition, feature extraction is complicated and it is difficult to maintain modal consistency; and the method based on conditional adaptive instance modulation may cause overfitting on small-scale cross-modal datasets. In addition, traditional deep convolutional neural networks have difficulty extracting modality-independent features when dealing with large cross-modal differences.

[0004] Compared with the modal invariant feature extraction method, the heterogeneous face generation method converts the face image from one modality to another by using conditional or unconditional image synthesis technology to match the face with the same modality. The heterogeneous face generation method has more direct image comparison ability, high-precision feature alignment, use of existing powerful models, enhanced data consistency, and high flexibility, thereby significantly improving the overall recognition effect and practicability. The early literature Q. Liu, X. Tang, H. Jin, H. Lu, and S. Ma, "A nonlinear approach for face sketch synthesis and recognition," in Proc. IEEE Comput. Soc. Conf. Comput. Vis. Pattern Recognit., San Diego, CA, USA, Jun. 2005, pp. 1005-1010. proposed an independent synthesis algorithm by dividing the face image into overlapping small blocks and reconstructing the small blocks with linear combination in the training set. However, due to the neglect of the compatibility between adjacent small blocks, the overall consistency of the generated face image is poor. The literature Fang, Y., Deng, W., Du, J., & Hu, J. (2020). Identity-aware CycleGAN for face photo-sketch synthesis and recognition. Pattern Recognit., 102, 107249. introduced the identity-aware CycleGAN model in the CycleGAN framework in the literature J. Zhu, T. Park, P. Isola, and A. A. Efros, "Unpaired Image-to-Image Translation Using Cycle-Consistent Adversarial Networks," in 2017 IEEE International Conference on Computer Vision (ICCV), 2017, pp. 2242-2251. The IACycleGAN model combines the bidirectional conversion generator and discriminator with the perception loss to improve the synthesis quality and retain the facial features for recognition.C. Fu, X. Wu, Y. Hu, H. Huang and R. He, "DVG-Face: Dual Variational Generation for Heterogeneous Face Recognition," in IEEE Transactions on Pattern Analysis and Machine Intelligence, 2022, pp. 2938-2952. proposed the DVG-Face model, which learns the joint distribution of paired heterogeneous images by introducing a dual generation mechanism in the variational autoencoder (VAE) framework to improve the synthesis quality and preserve facial features for recognition. The above documents all use two generators for bidirectional conversion of heterogeneous images, and ensure high-fidelity identity information through identity-preserving loss and contrastive learning mechanism, thereby enhancing the recognition performance. Although identity-aware supervision and identity-aware loss are introduced in these models respectively, there is still a problem of insufficient fusion of local features and global features, and a large amount of high-quality paired data is still needed for training. In the process of face modal conversion, the inability to correctly handle local details and global consistency will reduce the quality of the generated image, making it lack of realism and naturalness, and obvious visual defects and artifacts appear, thereby affecting the face recognition effect in actual application. SUMMARY

[0005] In view of the problems of insufficient fusion of local details and global features in the above recognition method based on heterogeneous face generation, the present application provides a heterogeneous face recognition method based on multi-feature fusion reconstruction generation. The method constructs a cycle generative adversarial network composed of a generator fused with a RSTB (Residual Swin Transformer Block) deep identity feature extractor and a discriminator based on RSTB, fuses and reconstructs the deep identity features, content features of hand-drawn faces and style features of visible light faces, realizes the modal conversion of hand-drawn faces to visible light faces, and performs heterogeneous face recognition based on the converted face pictures, which is conducive to improving the recognition performance.

[0006] Firstly, the application proposes a generator fused with an RSTB deep identity feature extractor. RSTB combines window block attention mechanism and residual connection, which can effectively perform parallel computing, accelerate the training and inference process of the model. The feature extractor has a hierarchical windowed attention mechanism, which can extract features at different scales and resolutions, which is beneficial to fuse local features and global features, so that the generated face picture can retain both local details and global consistency. At the same time, the RSTB deep feature extractor can effectively handle long-distance dependencies, helping the generator to understand the image structure globally and generate structurally realistic and reasonable real face images. Secondly, the application introduces new identity feature loss function and style feature loss function. The identity feature loss function directly calculates the difference between the generated visible light face and the hand-drawn face image to measure the preservation of identity features and speed up the convergence speed of identity features. The style feature loss function adds skewness calculation when calculating the difference between the generated image and the visible light style face image, preventing the model from excessively biasing the training dataset when generating faces, thereby improving the generalization ability of the model generated samples. Finally, the application constructs a discriminator model based on RSTB. RSTB can not only effectively extract local details of images, but also ensure the consistency of overall effects, fully fuse global features and local features. Using RSTB for feature extraction and discrimination enhances the diversity of contrast features, which is beneficial to improve the performance of the discriminator. The introduction of RSTB increases the complexity of the discriminator model, and the complexity of the generator model is balanced, thereby enhancing the stability of the training.

[0007] To achieve the above purpose, the application provides the following technical scheme:

[0008] A heterogeneous face recognition method based on multi-feature fusion reconstruction generation, the method comprising: obtaining a target face image, inputting the target face image into a trained generator to obtain a converted face image corresponding to the target face image; inputting the target face image and its corresponding converted face image into a trained classifier to obtain an identity recognition result corresponding to the target face image; wherein the training process comprises:

[0009] Step 1, obtaining a real face image pair and its corresponding identity label, the real face image pair being a hand-drawn face image and a visible light face image of the same object;

[0010] Step 2, input the real face image pair into the generator, obtain the content feature or the style feature through the VGG feature extractor, obtain the shallow identity feature through the shallow feature extractor, and obtain the deep identity feature by processing the shallow identity feature through the deep feature extractor; fuse the style feature of the hand-drawn face image of the same object with the deep identity feature and the content feature of the visible light face image, and obtain the reconstructed hand-drawn face image through the face reconstructor; fuse the style feature of the visible light face image of the same object with the deep identity feature and the content feature of the hand-drawn face image, and obtain the reconstructed visible light face image through the face reconstructor;

[0011] Step 3, input the real visible light face image and the reconstructed visible light face image into the discriminator, and obtain the discrimination result of whether the input visible light face image is the reconstructed visible light face image through the discriminator; input the real hand-drawn face image and the reconstructed hand-drawn face image into the discriminator, and obtain the discrimination result of whether the input hand-drawn face image is the reconstructed hand-drawn face image through the discriminator;

[0012] Step 4, input the reconstructed visible light face image or / and the reconstructed hand-drawn face image into the classifier, and obtain the identity prediction result of the input face image through the classifier;

[0013] Step 5, calculate the style loss value, the content loss value and the identity loss value of step 2, the discrimination result of step 3, the adversarial loss value of the real face image, and the cycle consistency loss value, the classification loss value of the identity prediction result of step 4 and the real identity label; return the loss value to update the training parameters until the training is completed.

[0014] Compared with the prior art, the method has the following advantages and beneficial effects:

[0015] 1) The cycle generative adversarial network model composed of the generator based on the RSTB deep identity feature extractor and the discriminator based on the deep identity feature extractor RSTB proposed in the application can effectively fuse the identity feature, the style feature, the content feature, the local feature and the global feature to generate a real face image with reasonable structure and identity preservation unsupervisedly, and the performance of face recognition can be effectively improved when the real face image is used for heterogeneous face recognition; 2) The identity feature loss and the style feature loss introduced in the application can accelerate the convergence speed of the model and further improve the generalization ability of the model. BRIEF DESCRIPTION OF DRAWINGS

[0016] In order to make the purpose, technical scheme and advantages of the application clearer, the application will be further described in detail below with reference to the drawings, in which:

[0017] Figure 1 It is a flow chart of the heterogeneous face recognition method in the embodiment of the application.

[0018] Figure 2 A heterogeneous face recognition model training flowchart in an embodiment of the present application;

[0019] Figure 3 A heterogeneous face recognition training schematic block diagram in an embodiment of the present application;

[0020] Figure 4 An RSTB network structure schematic diagram in an embodiment of the present application;

[0021] Figure 5 An STL network structure schematic diagram in an embodiment of the present application;

[0022] Figure 6 A generator network structure schematic diagram in an embodiment of the present application;

[0023] Figure 7 A heterogeneous face recognition model test schematic block diagram in an embodiment of the present application. DETAILED DESCRIPTION

[0024] The technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only some of the embodiments of the present application, but not all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative work fall within the scope of protection of the present application.

[0025] As shown in Figure 1 The heterogeneous face recognition method provided by the present embodiment combines feature decoupling and modal conversion, and the method comprises the following steps: obtaining a target face image, inputting the target face image into a trained generator to obtain a converted face image corresponding to the target face image; inputting the target face image and the converted face image corresponding thereto into a trained classifier to obtain an identity recognition result corresponding to the target face image.

[0026] In some embodiments of the present application, the heterogeneous face recognition method of the present embodiment is mainly used for type recognition of heterogeneous faces, such as recognition of face images including real faces, cartoon faces and hand-drawn faces. The target face image can be an image collected by a visible light camera under the authorization of a user, and a hand-drawn image called from a hand-drawn database. In actual application, live video in a specific scene is collected by a camera or a cluster monitoring device, and an image extraction software is used to recognize and mark the head in the live video. Based on the marking, the head is cut out from the picture of the video to generate a target face image. The target face image is input into the trained generator and classifier, and an identity recognition result corresponding to the target face image can be obtained.

[0027] As Figure 2 shown, the training process corresponding to the heterogeneous face recognition model includes:

[0028] Step 1, obtaining real face image pairs and their corresponding identity labels, the real face image pairs being hand-drawn face images and visible light face images of the same object;

[0029] In an embodiment of the present application, the images of the training set come from the CUFSF (CUHK Face Sketch FERET Database) dataset, including visible light face images of 606 people, each visible light face image having a matching hand-drawn face image, and the visible light face images and the matching hand-drawn face images all having corresponding identity labels capable of distinguishing face images of different objects. All images in the training set are subjected to Resize processing to adjust the size of all images to 128x128x3.

[0030] For example, at the beginning of training, N pairs of heterogeneous visible light face images and hand-drawn face images are selected from the training set, that is, 2N face images are selected, or N pairs of face images can also be selected, which can include heterogeneous face image pairs or face image pairs with different image attributes, that is, the number of visible light face images and hand-drawn face images does not completely correspond; this embodiment is described by selecting N pairs of heterogeneous face images, each pair of heterogeneous face images in the N pairs of heterogeneous face images corresponding to a target face in N different face objects, the face image with serial number a corresponding to the face image with serial number N+a corresponding to the same face object, the face image with serial number a being a visible light face image and the face image with serial number N+a being a hand-drawn face image, both having the same identity label a', which is used to identify the object identity of the face image.

[0031] Step 2, inputting the real face image pair into the generator, obtaining the content feature or the style feature through the VGG feature extractor, obtaining the shallow identity feature through the shallow feature extractor, and obtaining the deep identity feature by processing the shallow identity feature through the deep feature extractor; fusing the style feature of the hand-drawn face image of the same object with the deep identity feature and the content feature of the visible light face image to obtain a reconstructed hand-drawn face image through the face reconstructor; fusing the style feature of the visible light face image of the same object with the deep identity feature and the content feature of the hand-drawn face image to obtain a reconstructed visible light face image through the face reconstructor;

[0032] In an embodiment of the present application, in the training stage, for convenience of description, as Figure 3As shown, the example of converting a hand-drawn face image to a visible light face image is used to illustrate the process. The i-th visible light face image in the training set is used as an example. and hand-drawn face images Simultaneously, the input is fed into generator G. In the generator, The hand-drawn face image is processed by the VGG feature extractor and the deep identity feature extractor to obtain the content features f. c With deep identity characteristics f id . The style features f of the visible light face image are obtained through the VGG feature extractor. s Hand-drawn facial image set I C The i-th hand-drawn face image in Extracted content features f c Deep identity characteristics f id Compared with the i-th visible light face image I S Chinese style characteristics f s The fused input is fed into the face reconstructor to obtain the i-th hand-drawn face image after fusion and reconstruction. Corresponding visible light face image Similarly, the i-th visible light face image after fusion and reconstruction is obtained. Corresponding hand-drawn face image

[0033] In this embodiment of the invention, the RSTB added to the generator G has a hierarchical windowed attention mechanism, performing feature extraction at different scales and resolutions. This facilitates the fusion of local and global features, thereby ensuring that the generated face image maintains both detail and global consistency. This makes the deep feature extraction of input cross-modal faces more refined. RSTB is composed of STL (Swin Transformer Layer). The network structure of RSTB and STL is as follows: Figure 4 and Figure 5 As shown. The feature extractor VGG in generator G adopts a custom VGG19 network structure, and generator G uses the following... Figure 6 The generator network structure is shown.

[0034] Step 3: Input the real visible light face image and the reconstructed visible light face image into the discriminator, and obtain the discrimination result of whether the input visible light face image is a reconstructed visible light face image; input the real hand-drawn face image and the reconstructed hand-drawn face image into the discriminator, and obtain the discrimination result of whether the input hand-drawn face image is a reconstructed hand-drawn face image; in this embodiment of the invention, the image is compared with the i-th hand-drawn face image. Corresponding visible light face image The i-th hand-drawn face image generated by generator G corresponding visible light face image The input visible light face image is input into the discriminator D to determine whether the input visible light face image is a reconstructed synthetic face image; similarly, the input hand-drawn face image can be determined whether it is a reconstructed synthetic face image.

[0035] The discriminator D adopts a gradually increasing convolutional neural network, which makes the model more stable when training high-resolution images by gradually increasing the resolution. The RSTB module is added to the discriminator D for deep feature extraction, which improves the discrimination ability of the discriminator for generated images, and also balances the complexity of the discriminator model and the generator model, thereby enhancing the stability of the training. The discrete wavelet transform is also introduced in the discriminator D to check the spectral difference between the face images, which to some extent suppresses the phenomenon of artifacts in the generated visible light face images.

[0036] It should be noted that, since the present application can convert the visible light face image to the hand-drawn face image, and also can convert the hand-drawn face image to the visible light face image, the model can include two generators G X and G Y , two discriminators D X and D Y ; the generator G X can process the visible light face image, the generator G Y can process the hand-drawn face image, the discriminator D X can process the real visible light face image and the reconstructed visible light face image, and the discriminator D Y can process the real hand-drawn face image and the reconstructed hand-drawn face image.

[0037] Step 4, input the reconstructed visible light face image or / and the reconstructed hand-drawn face image into the classifier to obtain the identity prediction result of the input face image through the classifier;

[0038] In the embodiment of the present application, the real visible light face image and the reconstructed visible light face image are input into the classifier C to obtain the face feature of the input face image and predict the identity of the input face image; similarly, the real hand-drawn face image and the reconstructed hand-drawn face image are input into the classifier C to obtain the face feature of the input face image and predict the identity of the input face image.

[0039] In some embodiments, the classifier adopts a LightCNN network structure pre-trained on the MS-Cleleb-1M dataset. LightCNN (Light Convolutional Neural Networks) is a lightweight convolutional neural network, which is particularly suitable for tasks such as face recognition. The network structure is designed to reduce model parameters and computational complexity while maintaining high recognition performance. In other possible implementations, the classifier can also adopt existing conventional classification models, such as Bayesian classifiers, neural network classifiers, etc., which are not specifically limited by the present application.

[0040] Step 5, calculate the style loss value, content loss value and identity loss value of step 2, the adversarial loss value of the discrimination result of step 3 and the real face image and the cycle consistency loss value, and the classification loss value of the identity prediction result of step 4 and the real identity label; return the loss value to update the training parameters until the training is completed. In the embodiment of the present application, considering that the loss function is in addition to the commonly used adversarial loss function in the same type method, the present application further includes an identity loss function, a cycle consistency loss function, a style feature loss function, a content feature loss function and a classification loss function;

[0041] First, the style feature loss function and the content feature loss function are used to constrain the feature extraction process of the image, respectively, to promote the feature extractor to extract only the content features of the hand-drawn face image to maintain the content information and face feature information before conversion, and the style feature extractor to extract only the style feature information of the visible light face image, so that the converted face image retains its content information while having the style information of the visible light face. The content feature loss function, the style feature loss and the feature extraction overall loss function formula are defined as follows:

[0042]

[0043] L total =αL c +βL s (4)

[0044] wherein w j represents the weight of the jth layer feature map, represents the jth layer feature map of the ith reconstructed face image extracted by the VGG feature extractor, represents the jth layer feature map of the ith real face image extracted by the VGG feature extractor, if the reconstructed face image is a visible light face image, the real face image corresponds to a hand-drawn face image, if the reconstructed face image is a hand-drawn face image, the real face image corresponds to a visible light face image, in order to facilitate the representation, the above formula is taken as an example of the case of hand-drawn face image to visible light face image conversion to explain the corresponding loss function, μ(·) represents the mean, σ(·) represents the variance, γ(·) represents the skewness, formula (3) is the skewness formula, and ||·||2 represents the L2 norm.

[0045] It can be seen from formula (1) that the content loss function reflects the identity information difference between the output image and the input image by directly calculating the difference between each layer feature map. It can be seen from formula (2) that the style loss function evaluates the style difference between the images by calculating the L2 norm of the mean and variance and the skewness of the input style image The supplementary measurement of the style difference by the skewness is beneficial to make the style of the generated image closer to the style image, while avoiding overfitting of the model. Formula (4) is the weighted sum of the style loss and the identity loss, wherein μ and β are the loss coefficients of the identity loss and the style loss respectively for adjusting the feature proportion. The function of adjusting the bias of the image is achieved, and according to the experiment, α = 100000, β = 0.01.

[0046] In some preferred embodiments of the present application, a standard deviation weight w j is introduced for each layer feature map. j The introduction of the weight w j makes the high change area occupy a larger weight in the loss, so that the details in the face image can be better preserved, and the weight w j is the inverse of the standard deviation of the face image feature map, for weighting the difference. The feature map with large standard deviation (high change area) occupies a larger weight in the loss. The feature map with large standard deviation represents the area with more changes, which usually contains more details and information. By applying a larger weight to these areas, the loss function can better focus on these important areas, thereby better preserving these details when generating the image. Therefore, the calculation formula of the weight of the jth layer feature map is represented as:

[0047]

[0048] wherein φ j (I c ) represents the jth layer feature map of all real face images extracted by the VGG feature extractor; and σ represents the standard deviation.

[0049] In some preferred embodiments of the present application, on the basis of the above-mentioned embodiments, the feature map is controlled more finely, and a pixel-level weight W j (p) can be expressed as:

[0050] W j (p) = 1 - 0.1 · d M (L, P) (6)

[0051] where d M (L, P) represents the Manhattan distance between the pth pixel point and the center pixel point L. The pixel-level weight allows precise control of the importance of each pixel point during the generation process. This method can ensure that the pixel points in important feature areas (such as facial features) are more important, thereby enhancing the accuracy and details of the generated image in these key areas. Therefore, the updated content loss value is expressed as:

[0052]

[0053] where W j (p) represents the weight of the pth pixel point in the jth layer feature map, represents the feature value of the pth pixel point in the jth layer feature map of the i th reconstructed face image extracted by the VGG feature extractor, represents the feature value of the pth pixel point in the jth layer feature map of the i th real face image extracted by the VGG feature extractor; if the reconstructed face image is a visible light face image, the real face image corresponds to a hand-drawn face image, and if the reconstructed face image is a hand-drawn face image, the real face image corresponds to a visible light face image.

[0054] The cycle consistency loss function is a key component in the network, which ensures that the image generated by converting from one domain to another domain and returning to the original domain is as close to the original image as possible. The identity loss function is used in the network to encourage the generator to generate an image consistent with the input image when the input image is already from the target domain. The formula definitions of the cycle consistency loss function and the identity loss function are as follows:

[0055]

[0056] where E[·] is the expected value, and φ i(·) represents the i-th layer feature map, ||·||1 represents the L1 norm, and G(·) represents the generator. Formula (8) is an identity loss function, formula (9) and formula (10) are cycle consistency loss functions, and formula (11) is the sum of cycle consistency loss values. The identity loss value between the output image and the input hand-drawn face image is obtained by calculating the L1 norm, and is used for parameter updating. The difference between the output result generated by the generator G and the hand-drawn face image and the visible light face image is calculated respectively to keep the identity information and style information of the generated result image ;

[0057] Then, a classification loss function is followed, and a softmax cross-entropy loss function is used to constitute the classification loss function, wherein the input of the classifier is the visible light image I generated by the generator o . The classification loss updates the classifier and optimizes the generator at the same time, and the formula is as follows:

[0058]

[0059] wherein y ij represents the true label value of sample i in class j, and y

[0060] The adversarial loss adopts the adversarial loss function of the CycleGAN network, and the formula is as follows:

[0061]

[0062] wherein, represents the i-th hand-drawn face image corresponding to the generated visible light face image, represents the generated visible light face image predicted as the probability of being a synthetic image, and y

[0063] The total loss function is as follows:

[0064] L=λ id L id +λ cycle L cycle +λ total L total +λ adv L adv +λ clf L clf (14)

[0065] wherein λ id , λ cycle , λ total , λ adv , λclf represent the weight of the importance of the corresponding loss. Through a large number of experimental experiences, λ id , λ cycle , λ total , λ adv , λ clf are respectively set to 5, 10, 5, 1, 0.1.

[0066] Training the modules in the network alternately can make the modal conversion face generation task, the feature extraction task and the heterogeneous face recognition task optimize and improve each other. The extraction of style features and identity features makes the modal conversion face generation task can be completed smoothly, so that the generated face image has both the identity information of the hand-drawn face image and the style information of the visible light face image, and the generated face image data augmentation plus the extracted identity features fused into the heterogeneous face recognition classifier can also promote the improvement of the performance of the heterogeneous face recognition.

[0067] In some preferred embodiments, the present application also adds a test stage to the above training process, as shown in FIG. 5, in the test stage, there are two methods of qualitative test and quantitative test to verify the effect of the model. Figure 7

[0068] In the qualitative test, the hand-drawn face image to be converted and the visible light face image are input into the generator G, and the modal conversion face image can be obtained. The quality of the synthesized modal conversion face image can be observed directly to verify the effect of the generated visible light face image.

[0069] In the quantitative test, the face image generated by the generator G and the face data set to be detected with labels are respectively input into the trained classifier C, and the Rank recognition rate is calculated by calculating the difference between the feature maps, that is, the probability of the first successful search for the correct gallery image of the generated face image The greater the Rank recognition rate, the better the effect of the model.

[0070] Although the embodiments of the present application have been shown and described, it can be understood by those of ordinary skill in the art that various changes, modifications, replacements and variations can be made to these embodiments without departing from the principles and spirits of the present application, and the scope of the present application is defined by the appended claims and their equivalents.​

Claims

1. A heterogeneous face recognition method based on multi-feature fusion reconstruction generation, characterized in that, The method comprises: acquiring a target face image, inputting the target face image into a trained generator to obtain a converted face image corresponding to the target face image; and inputting the target face image and the converted face image corresponding thereto into a trained classifier to obtain an identity recognition result corresponding to the target face image; wherein the training process comprises: Step 1: acquiring a real face image pair and an identity label corresponding thereto, the real face image pair being a hand-drawn face image and a visible light face image of the same object; Step 2: inputting the real face image pair into the generator, obtaining a content feature or a style feature through a VGG feature extractor, obtaining a shallow identity feature through a shallow feature extractor, and obtaining a deep identity feature by processing the shallow identity feature through a deep feature extractor; fusing the style feature of the hand-drawn face image of the same object with the deep identity feature and the content feature of the visible light face image to obtain a reconstructed hand-drawn face image through a face reconstructor; fusing the style feature of the visible light face image of the same object with the deep identity feature and the content feature of the hand-drawn face image to obtain a reconstructed visible light face image through the face reconstructor; Step 3: inputting the real visible light face image and the reconstructed visible light face image into the discriminator to obtain a discrimination result of whether the input visible light face image is the reconstructed visible light face image through the discriminator; inputting the real hand-drawn face image and the reconstructed hand-drawn face image into the discriminator to obtain a discrimination result of whether the input hand-drawn face image is the reconstructed hand-drawn face image through the discriminator; Step 4: inputting the reconstructed visible light face image or / and the reconstructed hand-drawn face image into the classifier to obtain an identity prediction result of the input face image through the classifier; Step 5: calculating a style loss value, a content loss value and an identity loss value of step 2, a discrimination result of step 3, an adversarial loss value of the real face image, a cycle consistency loss value, an identity prediction result of step 4 and a real identity label, and returning the loss values to update the training parameters until the training is completed.

2. The heterogeneous face recognition method based on multi-feature fusion reconstruction generation of claim 1, characterized in that, The discriminator comprises a deep feature extractor, a discrete wavelet transform layer HarrTransform, an RGB image processing layer, a convolution layer and a linear layer.

3. The heterogeneous face recognition method based on multi-feature fusion reconstruction generation according to claim 1 or 2, characterized in that, The deep feature extractor comprises a plurality of STL layers and an LN normalization layer; each STL layer is sequentially connected, and an output end of the last STL layer is connected to an input end of the LN normalization layer; an input end of the first STL layer is superimposed with an output end of the LN normalization layer and serves as an output end of the next layer.

4. The heterogeneous face recognition method based on multi-feature fusion reconstruction generation of claim 1, characterized in that, The calculation formula of the content loss value is: wherein w j represents the weight of the jth layer feature map of the i th real face image, represents the jth layer feature map of the i th reconstructed face image extracted by the VGG feature extractor, represents the jth layer feature map of the i th real face image extracted by the VGG feature extractor; if the reconstructed face image is the visible light face image, the real face image corresponds to the hand-drawn face image, and if the reconstructed face image is the hand-drawn face image, the real face image corresponds to the visible light face image.

5. The heterogeneous face recognition method based on multi-feature fusion reconstruction generation according to claim 4, characterized in that, The calculation formula of the weight of the jth layer feature map is: where φ j (I c ) represents the jth layer feature map of all real face images extracted by the VGG feature extractor; and σ represents the standard deviation.

6. The heterogeneous face recognition method based on multi-feature fusion reconstruction generation of claim 1, characterized in that, The calculation formula of the content loss value is: wherein, W j (p) represents the weight of the pth pixel point in the jth layer feature map of the ith reconstructed face image, represents the feature value of the pth pixel point in the jth layer feature map of the ith reconstructed face image extracted by the VGG feature extractor, represents the feature value of the pth pixel point in the jth layer feature map of the ith real face image extracted by the VGG feature extractor; if the reconstructed face image is a visible light face image, the real face image corresponds to a hand-drawn face image, and if the reconstructed face image is a hand-drawn face image, the real face image corresponds to a visible light face image.

7. The heterogeneous face recognition method based on multi-feature fusion reconstruction generation according to claim 6, characterized in that, The calculation formula of the content loss value is: W j (p) = 1 - 0.1 · d M (L, P) wherein d M (L, P) denotes the Manhattan distance of the pth pixel point from the center pixel point L. 8.The heterogeneous face recognition method based on multi-feature fusion reconstruction generation of claim 1, characterized in that, The calculation formula of the identity loss value is: wherein E[·] is an expectation value, denotes the feature map of the j-th layer of the i-th reconstructed face image extracted by the deep feature extractor, denotes the feature map of the j-th layer of the i-th real face image extracted by the deep feature extractor, wherein if the reconstructed face image is a visible light face image, the real face image corresponds to the visible light face image, and if the reconstructed face image is a hand-drawn face image, the real face image corresponds to the hand-drawn face image. 9.The heterogeneous face recognition method based on multi-feature fusion reconstruction generation of claim 1, characterized in that, The calculation formula of the style loss value is: where N l denotes the number of feature maps, μ(·) denotes the mean, σ(·) denotes the variance, γ(·) denotes the skewness, denotes the j-th layer feature map of the i-th reconstructed face image extracted by the VGG feature extractor, denotes the j-th layer feature map of the i-th real face image extracted by the VGG feature extractor, ||·||2 denotes the L2 norm, if the reconstructed face image is a visible light face image, the corresponding real face image is a visible light face image, if the reconstructed face image is a hand-drawn face image, the corresponding real face image is a hand-drawn face image.

10. The heterogeneous face recognition method based on multi-feature fusion reconstruction generation of claim 9, characterized in that, The calculation formula of the skewness is: where φ j (I) denotes the feature map of the jth layer of all real face images extracted by the VGG feature extractor.

Images