Heterogeneous face recognition method based on instance-level spatial perception guidance

Through the instance-level spatial perception-guided heterogeneous face recognition method, the cross-spectrum face transformation network and semantic alignment module are used to solve the problem of low quality of near-infrared face image conversion, and high-precision natural face image generation is achieved, reducing the The cost of labeling is improved and the accuracy of facial recognition is improved.

CN114581975BActive Publication Date: 2024-11-05SOUTH CHINA UNIV OF TECH
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202210155433.9
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2022-02-21
Publication Date
2024-11-05
Estimated Expiration
2042-02-21

AI Technical Summary

Technical Problem

In the prior art, in heterogeneous face recognition, when the near-infrared face image is converted into a natural face image, the generated image quality is low, resulting in a decrease in recognition accuracy, and relying on additional face semantic segmentation annotation, which is expensive to apply.

Method used

Using a heterogeneous face recognition method based on instance-level spatial perception guidance, a cross-spectral face transformation network is designed through semantic spatial distribution adaptive alignment module and semantic alignment spatial adaptive normalization module, and a two-way consistent training framework combined with CycleGAN , using the structural consistency loss function of gradients, optimize the conversion of NIR images into high-quality VIS images, and reduce labeling dependencies.

Benefits of technology

The quality of natural face images generated by near-infrared image conversion is improved, the accuracy of face recognition is enhanced, the need for additional annotation is reduced, and the robustness and applicability of the model is improved.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN114581975B_ABST
    Figure CN114581975B_ABST
Patent Text Reader

Abstract

The present invention discloses a heterogeneous face recognition method based on instance-level spatial perception guidance, including: using an open source NIR-VIS heterogeneous face dataset; proposing a semantic space distribution adaptive alignment module and a semantic aligned spatial adaptive normalization module, designing a cross-spectral face illusion network that uses any VIS or NIR image as an example to guide the conversion between NIR and VIS images, and establishing an unsupervised NIR-VIS image translation model based on this network structure; based on the two-way consistent training framework of CycleGAN, proposing a gradient-based structural consistency loss function to optimize the model; using the optimized model to convert NIR images into VIS images for face recognition. The present invention improves the quality and diversity of NIR-VIS image conversion, improves the face recognition accuracy of near-infrared images, and the feature of arbitrary VIS instance-guided conversion improves the robustness of the model in different scenarios. Further, the present method can be seamlessly implanted into existing face recognition systems.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the technical field of face image recognition captured by different imaging sensors, and in particular to a heterogeneous face recognition method based on instance-level spatial perception guidance. Background Art

[0002] Face recognition technology is a technology that identifies people based on their facial features, and has important applications in security, monitoring, payment, access control, and detection. On the other hand, different systems use different imaging sensors. For example, near-infrared cameras can capture clear facial images under low-light conditions (such as at night), but the spectral differences based on imaging lead to a domain gap between near-infrared facial images (NIR images) and natural facial images (VIS images). Face recognition models built based on natural images cannot accurately extract the identity features of near-infrared facial images, resulting in reduced recognition accuracy.

[0003] Face recognition captured by different imaging sensors (heterogeneous face recognition) is a hot research topic in computer vision. The mainstream research methods can be divided into three strategies:

[0004] 1. Common latent space learning: Face images in different domains (imaged under different spectra) are projected into a common representation space, and the expression of identity features of face images is not affected by the spectrum.

[0005] 2. Domain-independent feature learning: Decompose the face image into domain-related parts and domain-independent parts, and learn a universal face recognition model based on domain-independent image features.

[0006] 3. Based on cross-domain image conversion: By converting face images imaged under other spectra into images under a specific spectrum (such as natural light), the heterogeneous face recognition problem is simplified to homogeneous face recognition.

[0007] The method of the present invention belongs to the third strategy. Among them, the earlier methods ignored the non-strict alignment of NIR and VIS images, modeled the NIR-VIS conversion problem as supervised image translation, or ignored the pathological nature of NIR image conversion to VIS image, which would lead to unnatural artifacts in the converted VIS image, low image quality, and unreality, resulting in reduced recognition accuracy. Current methods are generally based on unsupervised image translation modeling, focusing on using VIS samples to guide NIR image conversion, but global pooling of the appearance features encoded in the VIS image limits the full utilization of the appearance features, a large amount of facial details and textures are lost, and the quality of the generated VIS image needs to be further improved; and model training relies on additional facial semantic segmentation spectrum annotations, and the cost of applying the method to other scenarios is high. Summary of the invention

[0008] The purpose of the present invention is to overcome the shortcomings and deficiencies of the prior art, and proposes a heterogeneous face recognition method based on instance-level spatial perception guidance, which can use the appearance features of VIS images that retain spatial information to guide the conversion of NIR images to generate VIS images, improve the quality of generated VIS images, and thus improve the accuracy of face recognition; and based on the image gradient, ensure the consistency of the structure before and after the image conversion, avoid additional labeling work, promote the retention of image identity features, and further improve the accuracy of face recognition.

[0009] To achieve the above object, the technical solution provided by the present invention is: a heterogeneous face recognition method based on instance-level spatial perception guidance, comprising the following steps:

[0010] 1) Data preparation

[0011] The open source NIR-VIS heterogeneous face dataset CASIA NIR-VIS2.0 is used, which contains near-infrared face images (NIR images) and natural face images (VIS images) of multiple people. After pre-processing by labeling, aligning and cropping all face images, the training set and test set are divided according to the public protocol in the field. The model trained based on this training set can be directly applied to other open source datasets for testing.

[0012] 2) Model construction

[0013] Considering that the NIR and VIS images of the same person in the dataset are not strictly aligned, a semantic space distribution adaptive alignment module and a semantic alignment space adaptive normalization module are proposed. A cross-spectral face hallucination network ICFH is designed, which uses any VIS or NIR image as an example to guide the conversion between NIR and VIS images. Based on this network structure, an unsupervised NIR-VIS image translation model is established.

[0014] 3) Model parameter tuning

[0015] Based on the bidirectional consistent training framework of CycleGAN, a gradient-based structural consistency loss function is proposed to solve the local deformation problem before and after image conversion and enhance the identity feature retention of people in NIR images.

[0016] 4) Face recognition

[0017] In the testing phase, the optimized model is used to generate the corresponding VIS version for a given NIR image to be identified, taking each VIS image in the face library as a reference instance, and calculating the identity feature similarity with the VIS image in the face library to complete the identification.

[0018] Furthermore, in step 1), the model is trained using the open source heterogeneous face dataset CASIA NIR-VIS2.0, and the training set and the test set are divided according to the 10-fold training setting commonly used in the field; the model trained based on the training set can be directly used for testing on the test sets of the other two open source heterogeneous face datasets Oulu-CASIA NIR-VIS and BUAA-VisNir;

[0019] In order to eliminate the differences between images caused by image acquisition itself, including the effects of different camera heights, distances, and inclinations on imaging, all face images used in the training and testing phases are preprocessed using the following steps:

[0020] 1.1) Facial key point annotation: manually annotate 5 key points on the face of the person in the image, including: left eye center, right eye center, nose tip, left corner of mouth and right corner of mouth;

[0021] 1.2) Face alignment: Rotate and scale the image according to the marked key points so that the key points satisfy a specific geometric relationship, i.e., the line between the midpoints of the left and right eye centers and the midpoints of the left and right mouth corners is vertical and has a length of n1 pixels;

[0022] 1.3) Face region cropping: A face region with a fixed height and width of H×W is cropped from the aligned image, where the cropping frame is horizontally centered on the midpoint of the line connecting the centers of the left and right eyes and is n2 pixels away from the upper border.

[0023] Furthermore, in step 2), an unsupervised NIR-VIS image translation model is proposed, which integrates the following parts:

[0024] 2.1) Adaptive alignment of semantic space distribution

[0025] To process NIR images With VIS image In order to solve the spatial misalignment, we propose a semantic spatial distribution adaptive alignment module SSAA. Semantic spatial distribution adaptive alignment refers to examining the content characteristics of NIR images at the feature level. Each position and VIS image appearance features The semantic relevance of each position on the image is calculated, and the spatial distribution of the VIS image appearance features is adjusted to align with the semantic distribution of the NIR image content features based on the relevance. The whole process is defined as:

[0026]

[0027] In the formula, Refers to the real number field, conv 1×1 Refers to 1×1 convolution operation, softmax is the logistic regression function, is the appearance feature of the aligned VIS image, H and W represent the height and width of the image, h, w and c′ represent the height, width and number of channels of the feature, and subscripts c and a represent the content feature and appearance feature, respectively;

[0028] 2.2) Spatial Adaptive Normalization for Semantic Alignment

[0029] In order to integrate the appearance features of aligned VIS images into the content features of NIR images to guide the network to generate sufficiently realistic VIS images, a semantically aligned spatial adaptive normalization module SASAN is proposed; SASAN is based on the appearance features of aligned VIS images. The modulation parameters are predicted and the content features of the whitened NIR image are recolored, thereby realizing the migration of the appearance features of the VIS image to the content features of the NIR image. The whole process is defined as:

[0030]

[0031] In the formula, is the NIR content feature output by the convolution operation of the lth layer in the network, and Respectively represent the network for the lth layer according to The predicted modulation parameters, ⊙ represents element-wise multiplication, and Respectively The mean and standard deviation of each channel are calculated as follows:

[0032]

[0033] In the formula, the subscript index Representation feature In pixels The entire encoding vector at ;

[0034] 2.3) Example-guided Cross-spectral Face Hallucination Network

[0035] Combining steps 2.1) and 2.2), we designed an instance-guided cross-spectral face hallucination network ICFH. Based on this network structure, we designed a generative network for converting NIR images to VIS images. and the generative network for converting VIS images to NIR images

[0036] Generative network for NIR image to VIS image conversion It consists of 3 parts, including: NIR content encoder VIS appearance encoder With VIS decoderDec vis ,Right now Receiving I nir To input and output NIR image content features Receiving I vis To input and output VIS image appearance features Dec vis take over and As input, firstly, the aligned VIS image appearance features are obtained by SSAA in step 2.1). Then decode and Get I nir Corresponding VIS version The entire image conversion process can be described as:

[0037]

[0038] In order to fully transfer the appearance features of the VIS image, the decoder Dec vis The normalization layer in the residual network layer of is replaced by SASAN in step 2.2); Specifically, for each "normalization-ReLU-convolution" network layer, the operation process can be described as:

[0039]

[0040] In the formula, and are the input features and output features of the network layer respectively, and ReLU is the rectified linear unit;

[0041] Similarly, the generative network for converting VIS images to NIR images It consists of 3 parts, including: VIS content encoder NIR appearance encoder With NIR decoder Dec nir ,Right now Receive as I vis Input and output VIS image content features Receiving I nir Input and output NIR image appearance features Dec nir take over and As input, firstly, the aligned NIR image appearance features are obtained by SSAA similar to step 2.1). Then decode and Get I vis The corresponding NIR version The entire image conversion process can be described as:

[0042]

[0043] In order to fully transfer the appearance features of the NIR image, the decoder Dec nir The normalization layer in the residual network layer of is replaced by SASAN similar to step 2.2); Specifically, for each "normalization-ReLU-convolution" network layer, the operation process can be described as:

[0044]

[0045] In the formula, and are the input and output features of the network layer, Γ nir With B nir Respectively according to The predicted modulation parameters of this network layer, and They are Channel-by-channel mean and standard deviation;

[0046] Finally, the designed unsupervised NIR-VIS image translation model consists of two pairs of generative adversarial networks, which are responsible for modeling the two mapping directions respectively; the direction from NIR to VIS conversion: the generative network VIS Image I vis To guide the NIR image I nir Convert to VIS version Identify the network Determine whether the image fed is Generated Or the real I vis ; Convert from VIS to NIR to generate network Taking NIR image I nir To boot, place the VIS image I vis Convert to NIR version Identify the network Determine whether the image fed is Generated Or the real I nir ;in, and The structure of can be any commonly used identification network.

[0047] Further, in step 3), let i and j represent any two persons, and are a NIR image and a VIS image of i respectively; and are a NIR image and a VIS image of j respectively; let’s consider Generate for boot VIS version by Generate for boot NIR version Model parameter tuning includes the following steps:

[0048] 3.1) Bidirectional Consistent Training Framework

[0049] The unsupervised NIR-VIS image translation model is optimized using the CycleGAN bidirectional consistency training framework, including the cycle consistency loss function. Adversarial Loss Function

[0050] Cycle consistency loss function Request to generate or Mapping in the opposite direction or And the original image or Provides appearance features as a guide to reconstruct the original image or The specific definition is:

[0051]

[0052] In the formula, and Respectively represent the NIR image set and VIS image set in the training data set, and Respectively represent the data sets and Medium Sampling Sample and represents the expected value of the loss function calculated for all sampled samples, and ||·||2 represents the L2 norm;

[0053] Adversarial Loss Function The ability of the generative network to generate realistic images is improved by confronting the generative network with the discriminative network. The specific definition is:

[0054]

[0055] In the formula, is a discriminant network that discriminates between the generated VIS image and the real VIS image, It is a discriminative network that discriminates the generated NIR images from the real NIR images;

[0056] 3.2) Identity feature preservation loss function

[0057] In order to keep the converted image with the same identity features as the original image, with the help of a pre-trained face recognition network Φ, the identity feature preservation loss function It is required that the two images are as close as possible in the representation space of the Φ projection, which is specifically defined as:

[0058]

[0059] In the formula, ||·||1 represents the L1 norm;

[0060] 3.3) Gradient-based structural consistency loss function

[0061] Gradient-based structural consistency loss function Require and and The structure remains unchanged; specifically, the Laplacian operator lap is used to extract the edge structure of the image, and the ReLU operator is used to suppress negative values ​​to obtain a sparse structural representation. The specific definition is as follows:

[0062]

[0063] In the formula, * represents convolution operation, and ReLU is the rectified linear unit;

[0064] The complete optimization objective of the unsupervised NIR-VIS image translation model is expressed as the weighted sum of the above-mentioned cycle consistency loss function, adversarial loss function, identity feature preservation loss function, and gradient-based structural consistency loss function. The model parameters are updated using the gradient backpropagation method; and the model is trained using the CASIANIR-VIS2.0 training set.

[0065] Further, in step 4), after the model training is completed, the generative network for converting NIR images to VIS images is taken. In the testing phase, take any NIR image to be recognized from the CASIANIR-VIS2.0 test set or other open source heterogeneous face recognition datasets Traverse each VIS image in the face library Gallery K is the number of VIS images, Boot as an instance Generate the corresponding VIS version Use the pre-trained face recognition network Φ to extract and Identity characteristics and Calculate cosine similarity The recognition result can be determined by locating the maximum cosine similarity, that is:

[0066] k * = argmax k {r (1) ,r (2) ,...,r (K)}

[0067] In the formula, k * That is The recognition result.

[0068] Compared with the prior art, the present invention has the following advantages and beneficial effects:

[0069] 1. The method of the present invention can make full use of the appearance features of the VIS image as a guiding example, retain the diversity of the features in spatial distribution, improve the quality and authenticity of the VIS image generated by NIR image conversion, and improve the face recognition accuracy of near-infrared images.

[0070] 2. The method of the present invention allows the use of any single VIS face image as an example to guide NIR image conversion, thereby improving the robustness of the model on different face databases and face recognition systems.

[0071] 3. The method of the present invention can fully transfer the appearance features of the guiding instance (including: lighting, skin color, background), and different VIS instances can guide the same NIR image to be converted into facial images with diverse appearances.

[0072] 4. The method of the present invention can directly use the existing natural face image recognition model for recognition, and thus can be seamlessly embedded in the existing face recognition system. BRIEF DESCRIPTION OF THE DRAWINGS

[0073] Figure 1 The figure is a logical flow diagram of the method of the present invention.

[0074] Figure 2 The generation network of NIR image to VIS image proposed by the method of the present invention Structural diagram.

[0075] Figure 3 The generation network of VIS image to NIR image proposed by the method of the present invention Structural diagram.

[0076] Figure 4 Schematic diagram of the structure of the spatial adaptive normalization module SASAN for semantic alignment proposed in the method of the present invention.

[0077] Figure 5 Schematic diagram of the residual block structure combined with SASAN.

[0078] Figure 6 The figure is a schematic diagram of an example of realizing near-infrared face image recognition by the method of the present invention. DETAILED DESCRIPTION

[0079] The present invention is further described in detail below in conjunction with embodiments and drawings, but the embodiments of the present invention are not limited thereto.

[0080] like Figure 1 As shown, the heterogeneous face recognition method based on instance-level spatial perception guidance provided in this embodiment includes the following specific implementation steps:

[0081] 1) Data preparation.

[0082] Download the CASIA NIR-VIS2.0 heterogeneous face dataset, divide it into training set and test set according to the 10-fold training setting, and also download the heterogeneous face datasets Oulu-CASIA NIR-VIS and BUAA-VisNir, and take their test sets.

[0083] For all face images, five key points of the face in the image are manually annotated, including: left eye center, right eye center, nose tip, left corner of the mouth and right corner of the mouth.

[0084] The marked key points are used to align the face. The specific operation is to rotate and scale the image so that the key points meet a specific geometric relationship, that is, the line connecting the midpoints of the left and right eye centers and the midpoints of the left and right mouth corners remains vertical and has a length of 43.2 pixels.

[0085] A face region with a fixed height of 128×128 is cropped from the aligned image, where the cropping frame satisfies the horizontal center of the midpoint of the line connecting the centers of the left and right eyes after alignment and is 40 pixels away from the upper border.

[0086] Randomly crop to get a near-infrared face image (i.e., NIR image) Natural face image (VIS image) in, Refers to the field of real numbers.

[0087] 2) Model construction.

[0088] 2.1) Adaptive alignment of semantic space distribution

[0089] like Figure 2 As shown, Semantic Spatially Adaptive Aligned Module (SSAA), NIR content encoder Extraction I nir Content Features VIS appearance encoder Extraction I vis Appearance characteristics The two features are each subjected to independent 1×1 convolution conv 1×1 Adjustment: Calculate the semantic correlation between each position on the NIR image content feature and each position on the VIS appearance feature, and adjust the spatial distribution of the VIS appearance feature to align with the semantic distribution of the NIR content feature based on the correlation. The process is described as follows:

[0090]

[0091] In the formula, softmax is the logistic regression function, is the VIS appearance feature after alignment.

[0092] like Figure 3 As shown, Semantic Spatially Adaptive Aligned Module (SSAA), VIS content encoder Extraction I vis Content Features NIR appearance encoder Extraction I nir Appearance characteristics The two features are each subjected to independent 1×1 convolution conv 1×1 Adjustment: Calculate the semantic correlation between each position on the VIS image content feature and each position on the NIR appearance feature, and adjust the spatial distribution of the NIR appearance feature to align with the semantic distribution of the VIS content feature based on the correlation. The process is described as follows:

[0093]

[0094] In the formula, is the aligned NIR appearance feature.

[0095] 2.2) Spatial Adaptive Normalization for Semantic Alignment

[0096] like Figure 4 As shown, in order to integrate the aligned VIS appearance features into the NIR content features to guide the network To generate sufficiently realistic VIS images, the semantically aligned spatial adaptive normalization module (SASAN: Semantic Aligned Spatial Adaptive Normalization) is used to DecoderDec vis In general, the convolution layer number of a certain residual block is denoted as l. is the NIR content feature of its output, and its channel-by-channel mean is calculated With standard deviation as follows:

[0097]

[0098] In the formula, the subscript index : indicates feature extraction In pixels The entire encoding vector at .

[0099] Use independent 1×1 convolutions according to Predicting modulation parameters and use and right For whitening, use and The whole process of recoloring the whitened NIR content features is as follows:

[0100]

[0101] Where ⊙ represents element-by-element multiplication.

[0102] Similarly, in order to integrate the aligned NIR appearance features into the VIS content features to guide the network Generate sufficiently realistic NIR images, the semantically aligned spatially adaptive normalization method works on DecoderDec nir In general, the convolutional layer in a residual block is numbered m. is the VIS content feature of its output, and its channel-by-channel mean is calculated With standard deviation as follows:

[0103]

[0104] In the formula, the subscript index x, y,: represents the feature The entire encoded vector at pixel (x,y).

[0105] Use independent 1×1 convolutions according to Predicting modulation parameters and use and right For whitening, use and The whitened VIS content features are recolored, and the whole process is shown as follows:

[0106]

[0107] Where ⊙ represents element-by-element multiplication.

[0108] 2.3) Example-guided Cross-spectral Face Hallucination Network

[0109] Combining steps 2.1) and 2.2), we designed an instance-guided cross-spectral face hallucination network ICFH. Based on this network structure, we designed a generative network for converting NIR images to VIS images. and the generative network for converting VIS images to NIR images

[0110] like Figure 2 As shown, the generative network for converting NIR images to VIS images It consists of 3 parts, including: NIR content encoder VIS appearance encoder With VIS decoderDec vis ,Right now Receiving I nir Input and output NIR content features Receiving I vis To input and output VIS appearance features VIS DecoderDec vis By 8 such Figure 5 The SASAN residual block and the two structures shown are stacked convolution blocks of "up-sampling-convolution-level normalization-ReLU", where: Figure 5 middle Represents element-by-element addition. vis take over and As input, the aligned VIS appearance features are obtained through the SSAA module The VIS appearance features are adaptively migrated to the NIR features through the SASAN module of each residual block; the original NIR image I is finally decoded. nir Corresponding VIS version The whole process above can be described as:

[0111]

[0112] Among them, for each "normalization-ReLU-convolution" network layer in each residual block, the operation process can be described as:

[0113]

[0114] In the formula, and They are the input features and output features of the network layer respectively, and ReLU is the rectified linear unit.

[0115] like Figure 3 As shown, the generative network for converting VIS images to NIR images It consists of 3 parts, including: VIS content encoder NIR appearance encoder With NIR decoder Dec nir ,Right now Receiving I vis To import and export VIS content features Receiving I nir Input and output NIR appearance features NIR DecoderDec nir By 8 similar Figure 5 The SASAN residual block and the two structures shown are stacked convolution blocks of "upsampling-convolution-level normalization-ReLU". nir take over and As input, the aligned NIR appearance features are obtained through SSAA alignment operation The SASAN method of each residual block is used to adaptively migrate the NIR appearance features to the VIS features; the original VIS image I is finally decoded. vis The corresponding NIR version The whole process above can be described as:

[0116]

[0117] Among them, for each "normalization-ReLU-convolution" network layer in each residual block, the operation process can be described as:

[0118]

[0119] In the formula, and are the input features and output features of the network layer respectively.

[0120] 2.4) Unsupervised NIR-VIS Image Translation Model

[0121] This embodiment is based on the network and An unsupervised NIR-VIS image translation model is designed. The model consists of two pairs of generative adversarial networks, which are responsible for modeling two mapping directions: the direction of NIR to VIS conversion and the direction of generative network VIS Image I vis To guide the NIR image I nir Convert to VIS version Identify the network Determine whether the image fed is Generated Or the real I vis ; Convert from VIS to NIR to generate network Taking NIR image I nir To boot, place the VIS image I vis Convert to NIR version Identify the network Determine whether the image fed is Generated Or the real I nir .in, and Using the same network structure, i.e., a multi-scale discriminator network, each discriminator consists of a global discriminator sub-network and a local discriminator sub-network, which are composed of 7 and 5 "spectral normalization-convolution-LeakyReLU" network layers respectively.

[0122] 3) Parameter Tuning

[0123] In order to maintain the consistency of the method, that is, in the image conversion process, the guiding instance and the image to be converted can come from any person, we restate some symbols as follows: i and j represent any two people (not necessarily the same person), and are a NIR image and a VIS image of i respectively; and are a NIR image and a VIS image of j respectively. Without loss of generality, consider guide generate Corresponding VIS version use guide generate Corresponding NIR version Model parameter tuning includes the following steps:

[0124] 3.1) Bidirectional Consistent Training Framework

[0125] This embodiment uses the bidirectional consistency training framework of CycleGAN to optimize the unsupervised NIR-VIS image translation model, and calculates the cycle consistency loss function and the adversarial loss function respectively.

[0126] Cycle consistency loss function Request to generate (or ) after mapping in the opposite direction (or ) and use the original image (or ) as a guide to provide appearance features and reconstruct the original image (or ), specifically defined as:

[0127]

[0128] In the formula, and Respectively represent the NIR image set and VIS image set in the training data set, and Respectively represent the data sets and Medium Sampling Sample and It represents the expected value of the loss function calculated for all sampled samples, and ||·||2 represents the L2 norm.

[0129] Adversarial Loss Function The ability of the generating network to generate realistic images is improved by the confrontation between the generating network and the discriminating network. In this embodiment, Including global adversarial loss function With the local adversarial loss function We measure the adversarial relationship between the generator network and the global discriminator sub-network and between the generator network and the local discriminator sub-network respectively. The specific definition is:

[0130]

[0131] In the formula, and They are the global identification sub-network and the local identification sub-network for identifying the generated VIS image and the real VIS image, and They are the global identification subnetwork and the local identification subnetwork for identifying the generated NIR images and the real NIR images respectively.

[0132] 3.2) Identity feature preservation loss function

[0133] Using the pre-trained face recognition network LightCNN29Φ, the identity feature preservation loss function requires that the two images are as close as possible in the representation space of the Φ projection, which is specifically defined as:

[0134]

[0135] Where ||·||1 represents the L1 norm.

[0136] 3.3) Gradient-based structural consistency loss function

[0137] Gradient-based structural consistency loss function The image structure is required to remain unchanged before and after transformation. Specifically, for NIR images The corresponding VIS image or VIS image The corresponding NIR image The Laplacian operator lap is used to extract the edge structure of the image, and the ReLU operator is used to suppress negative values ​​to obtain a sparse structural representation. The structural representations of the corresponding image pairs should be as similar as possible, as described below:

[0138]

[0139] In the formula, * represents the convolution operation.

[0140] The optimization objective of the entire unsupervised NIR-VIS image translation model is expressed as the weighted sum of the above-mentioned cycle consistency loss function, adversarial loss function, identity feature preservation loss function, and gradient-based structural consistency loss function, with weights of 10, 1.0, 0.5, and 1.0, respectively. The model is iterated and updated 1 million times. In addition, the model parameters are optimized using the gradient optimization method with Adam adaptive momentum evaluation, where the learning rate and decay coefficient are both set to 0.0001, the learning rate decays linearly to 0.0 after 500,000 iterations, and the minimum batch size is set to 1. The training and testing of the model are implemented based on Pytorch1.7.0 and completed on a single Rtx3090 graphics card.

[0141] 4) Face recognition.

[0142] like Figure 6 As shown, the model training is completed, and the generative network for converting NIR images to VIS images is taken It is used for testing on the test sets of CASIA NIR-VIS2.0, Oulu-CASIA NIR-VIS and BUAA-VisNir. Taking the test set of CASIA NIR-VIS2.0 as an example, the known face library Gallery contains VIS images of K people. Right now:

[0143] For any NIR image to be identified in the test set Traverse each VIS template in the face library by Bootstrap for Instance Generate the corresponding natural image version Use the pre-trained face recognition network LightCNN29Φ to extract and Identity characteristics and Calculate cosine similarity

[0144] The final recognition result can be determined by locating the maximum cosine similarity, that is:

[0145] k * = argmax k {r (1) ,r (2) ,...,r (K)}

[0146] Among them, k * That is The recognition result.

[0147] The above embodiments are preferred implementation modes of the present invention, but the implementation modes of the present invention are not limited to the above embodiments. Any other changes, modifications, substitutions, combinations, and simplifications that do not deviate from the spirit and principles of the present invention should be equivalent replacement methods and are included in the protection scope of the present invention.

Claims

1. A heterogeneous face recognition method based on instance-level spatial perception guidance, characterized in that: The following steps are involved: 1) Data preparation The open source NIR-VIS heterogeneous face dataset CASIA NIR-VIS2.0 is used, which contains near-infrared face images of multiple people, namely NIR images, and natural face images, namely VIS images; After preprocessing all face images by labeling, aligning and cropping, they are divided into training and test sets according to public protocols in the field. The model is trained using the open source heterogeneous face dataset CASIA NIR-VIS2.0, and the training set and test set are divided according to the public 10-fold training setting in the field; the model trained based on this training set can be directly used for testing on the test sets of two other open source heterogeneous face datasets Oulu-CASIA NIR-VIS and BUAA-VisNir; In order to eliminate the differences between images caused by image acquisition itself, including the effects of different camera heights, distances, and inclinations on imaging, all face images used in the training and testing phases are preprocessed using the following steps: 1.1) Facial key point annotation: manually annotate 5 key points on the face of the person in the image, including: left eye center, right eye center, nose tip, left corner of mouth and right corner of mouth; 1.2) Face alignment: Rotate and scale the image according to the marked key points so that the key points satisfy a specific geometric relationship, i.e., the line between the midpoints of the left and right eye centers and the midpoints of the left and right mouth corners is vertical and has a length of n1 pixels; 1.3) Face region cropping: crop a face region with a fixed height and width of H×W from the aligned image, where the cropping frame is horizontally centered on the midpoint of the line connecting the left and right eye centers and is n2 pixels away from the upper border; 2) Model construction Considering that the NIR and VIS images of the same person in the dataset are not strictly aligned, a semantic space distribution adaptive alignment module and a semantic alignment space adaptive normalization module are proposed. A cross-spectral face hallucination network ICFH is designed, which uses any VIS or NIR image as an example to guide the conversion between NIR and VIS images. Based on this network structure, an unsupervised NIR-VIS image translation model is established. An unsupervised NIR-VIS image translation model is proposed, which integrates the following parts: 2.1) Adaptive alignment of semantic space distribution To process NIR images With VIS image In order to solve the spatial misalignment, we propose a semantic spatial distribution adaptive alignment module SSAA. Semantic spatial distribution adaptive alignment refers to examining the content characteristics of NIR images at the feature level. Each position and VIS image appearance features The semantic relevance of each position on the image is calculated, and the spatial distribution of the VIS image appearance features is adjusted to align with the semantic distribution of the NIR image content features based on the relevance. The whole process is defined as: In the formula, Refers to the real number field, conv 1×1 Refers to 1×1 convolution operation, softmax is the logistic regression function, is the appearance feature of the aligned VIS image, H and W represent the height and width of the image, h, w and c′ represent the height, width and number of channels of the feature, and subscripts c and a represent the content feature and appearance feature, respectively; 2.2) Spatial Adaptive Normalization for Semantic Alignment In order to integrate the appearance features of aligned VIS images into the content features of NIR images to guide the network to generate sufficiently realistic VIS images, a semantically aligned spatial adaptive normalization module SASAN is proposed; SASAN is based on the appearance features of aligned VIS images. The modulation parameters are predicted and the content features of the whitened NIR image are recolored, thereby realizing the migration of the appearance features of the VIS image to the content features of the NIR image. The whole process is defined as: In the formula, is the NIR content feature output by the convolution operation of the lth layer in the network, and Respectively represent the network for the lth layer according to The predicted modulation parameters, ⊙ represents element-wise multiplication, and Respectively The mean and standard deviation of each channel are calculated as follows: In the formula, the subscript index Representation feature In pixels The entire encoding vector at ; 2.3) Example-guided Cross-spectral Face Hallucination Network Combining steps 2.1) and 2.2), we designed an instance-guided cross-spectral face hallucination network ICFH. Based on this network structure, we designed a generative network for converting NIR images to VIS images. and the generative network for converting VIS images to NIR images Generative network for NIR image to VIS image conversion It consists of 3 parts, including: NIR content encoder VIS appearance encoder With VIS decoderDec vis ,Right now Receiving I nir To input and output NIR image content features Receiving I vis To input and output VIS image appearance features Dec vis take over and As input, firstly, the aligned VIS image appearance features are obtained by SSAA in step 2.1) Then decode and Get I nir Corresponding VIS version The entire image conversion process can be described as: In order to fully transfer the appearance features of the VIS image, the decoder Dec vis The normalization layer in the residual network layer of is replaced by SASAN in step 2.2); Specifically, for each "normalization-ReLU-convolution" network layer, the operation process can be described as: In the formula, and are the input features and output features of the network layer respectively, and ReLU is the rectified linear unit; Similarly, the generative network for converting VIS images to NIR images It consists of 3 parts, including: VIS content encoder NIR appearance encoder With NIR decoder Dec nir ,Right now Receive as I vis Input and output VIS image content features Receiving I nir Input and output NIR image appearance features Dec nir take over and As input, firstly, the aligned NIR image appearance features are obtained by SSAA similar to step 2.1). Then decode and Get I vis The corresponding NIR version The entire image conversion process can be described as: In order to fully transfer the appearance features of the NIR image, the decoder Dec nir The normalization layer in the residual network layer of is replaced by SASAN similar to step 2.2); Specifically, for each "normalization-ReLU-convolution" network layer, the operation process can be described as: In the formula, and are the input and output features of the network layer, Γ nir With B nir Respectively according to The predicted modulation parameters of this network layer, and They are Channel-by-channel mean and standard deviation; Finally, the designed unsupervised NIR-VIS image translation model consists of two pairs of generative adversarial networks, which are responsible for modeling the two mapping directions respectively; the direction from NIR to VIS conversion: the generative network VIS Image I vis To guide the NIR image I nir Convert to VIS version Identify the network Determine whether the image fed is Generated Or the real I vis ; Convert from VIS to NIR to generate network Taking NIR image I nir To boot, place the VIS image I vis Convert to NIR version Identify the network Determine whether the image fed is Generated Or the real I nir ;in, and The structure of can be any commonly used identification network; 3) Model parameter tuning Based on the bidirectional consistent training framework of CycleGAN, a gradient-based structural consistency loss function is proposed to solve the local deformation problem before and after image conversion and enhance the identity feature retention of people in NIR images. Let i and j represent any two persons. and are a NIR image and a VIS image of i respectively; and are a NIR image and a VIS image of j respectively; let’s consider Generate for boot VIS version by Generate for boot NIR version Model parameter tuning includes the following steps: 3.1) Bidirectional Consistent Training Framework The unsupervised NIR-VIS image translation model is optimized using the CycleGAN bidirectional consistency training framework, including the cycle consistency loss function. Adversarial Loss Function Cycle consistency loss function Request to generate or Mapping in the opposite direction or And the original image or Provides appearance features as a guide to reconstruct the original image or The specific definition is: In the formula, and Respectively represent the NIR image set and VIS image set in the training data set, and Respectively represent the data sets and Medium Sampling Sample and represents the expected value of the loss function calculated for all sampled samples, and ||·||2 represents the L2 norm; Adversarial Loss Function The ability of the generative network to generate realistic images is improved by confronting the generative network with the discriminative network. The specific definition is: In the formula, is a discriminant network that discriminates between the generated VIS image and the real VIS image, It is a discriminative network that discriminates the generated NIR images from the real NIR images; 3.2) Identity feature preservation loss function In order to keep the converted image with the same identity features as the original image, with the help of a pre-trained face recognition network Φ, the identity feature preservation loss function It is required that the two images are as close as possible in the representation space of the Φ projection, which is specifically defined as: In the formula, ||·||1 represents the L1 norm; 3.3) Gradient-based structural consistency loss function Gradient-based structural consistency loss function Require and and The structure remains unchanged; specifically, the Laplacian operator lap is used to extract the edge structure of the image, and the ReLU operator is used to suppress negative values ​​to obtain a sparse structural representation. The specific definition is as follows: In the formula, * represents convolution operation, and ReLU is the rectified linear unit; The complete optimization objective of the unsupervised NIR-VIS image translation model is expressed as the weighted sum of the above-mentioned cycle consistency loss function, adversarial loss function, identity feature preservation loss function, and gradient-based structural consistency loss function. The model parameters are updated using the gradient backpropagation method. The unsupervised NIR-VIS image translation model is trained using the CASIA NIR-VIS2.0 training set. 4) Face recognition In the testing phase, the optimized model is used to generate the corresponding VIS version for a given NIR image to be identified, taking each VIS image in the face database as a reference instance, and calculating the identity feature similarity with the VIS image in the face database to complete the identification; After the model training is completed, the generative network for converting NIR images to VIS images is taken In the testing phase, take any NIR image to be recognized from the CASIANIR-VIS2.0 test set or other open source heterogeneous face recognition datasets Traverse each VIS image in the face library Gallery K is the number of VIS images, Boot as an instance Generate the corresponding VIS version Use the pre-trained face recognition network Φ to extract and Identity characteristics Calculate cosine similarity The recognition result can be determined by locating the maximum cosine similarity, that is: In the formula, k * That is The recognition result.