Dual-coding face synthesis method and system for infrared-visible light face recognition

Through the dual encoding structure of identity encoder and style encoder, combining infrared and visible face information, the problems of unclear details and poor identity retention in cross-modal face synthesis are solved, and the accuracy of infrared-visible face recognition is improved and robustness.

CN118629081BActive Publication Date: 2024-11-05EAST CHINA JIAOTONG UNIVERSITY
View PDF 9 Cites 0 Cited by

Patent Information

Application Number
CN202411107294.8
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2024-08-13
Publication Date
2024-11-05
Estimated Expiration
2044-08-13

AI Technical Summary

Technical Problem

The existing infrared-visible face recognition technology faces the lack of large-scale paired data during cross-modal face synthesis, resulting in insufficient image details, changes in facial attributes and artifacts, making it difficult to maintain identity characteristics. Impacts recognition accuracy.

Method used

Using the dual-encoding structure of identity encoder and style encoder, the identity feature encoding is obtained through infrared face photos, combined with the style information of visible face photos for fusion and enhancement, and a jump connection is established in the decoder to guide the generation of more realistic and clear visible light face image.

Benefits of technology

It improves the authenticity and recognition accuracy of infrared image conversion generation, enhances the robustness of model, ensures the clarity and identity retention of facial features, and can effectively distinguish facial contours and details, and is suitable for different face libraries and systems.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118629081B_ABST
    Figure CN118629081B_ABST
Patent Text Reader

Abstract

The present invention proposes a dual-coding face synthesis method and system for infrared-visible light face recognition, which adopts a dual-coding structure of an identity encoder and a style encoder, takes an infrared face photo as a real photo, inputs the real photo into the identity encoder, and obtains the identity feature code; takes a random visible light face photo as a reference photo, inputs the reference photo and the real photo into the style encoder, and obtains the content code with the style information of the reference photo; fuses and enhances the content code and the identity feature code to obtain the enhanced features; inputs the enhanced features into the style embedded decoder, and establishes jump connections of each layer between the style encoder and the decoder to pass the feature information captured by the style encoder to the style embedded decoder, thereby guiding the generation of decoding features. The method of the present invention utilizes the prior information of the reference photo in the target domain to improve the authenticity of the VIS image generated by the NIR image conversion, and improves the accuracy of NIR face recognition.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the field of computer image processing, and in particular to a dual-coded face synthesis method and system for infrared-visible light face recognition. Background Art

[0002] As a non-contact, non-mandatory biometric technology, face recognition technology has attracted widespread attention in the field of computer vision and pattern recognition. In recent years, with the rapid development of deep learning, it has provided strong technical support for improving the accuracy of face recognition. However, the photos collected during face recognition are mainly images in visible light (VIS) scenes, and the recognition equipment has poor recognition effect on face images in non-visible light modalities. In real life, different sensor devices are often used to capture faces. For example, near-infrared (NIR) sensors are widely used in authentication systems and video surveillance cameras, which has led to the demand for heterogeneous face recognition.

[0003] Methods for heterogeneous face synthesis can be divided into three categories: methods based on domain-invariant features, common methods based on subspace learning, and methods based on image synthesis. Among them, methods based on image synthesis, also known as cross-modal face synthesis, refer to the conversion of images from one modality to another to identify faces in the same modality. Due to the low cost and high efficiency of end-to-end image synthesis using deep neural networks, cross-modal face synthesis technology has flourished, but these synthesis methods face several major challenges: (1) There is a lack of large-scale paired data for supervised modeling in actual scenarios, making it difficult to use pixel-level constraints to improve the quality of face image synthesis; (2) There are significant differences in texture and geometric details between classes due to the serious sensing gap between sensing devices between different modal domains, making the synthesized faces lack facial details and too smooth; (3) In the same modal domain, faces have significant intra-class differences due to differences in lighting, accessories, posture, skin color, etc., making face matching more difficult; (4) Affected by incompletely matched images, synthesized faces have artifacts and deformations, and identity preservation cannot be guaranteed.

[0004] In order to effectively solve the above problems, in the past decade, many methods for cross-modal face synthesis have been proposed. With the continuous improvement of face synthesis quality, under the guidance of the "recognition by generation" strategy, the performance of face recognition technology has also been continuously improved. For example, CycleGAN, DVG_Face, CoConGAN is a few-shot, unsupervised synthesis framework, which proposes a cooperative contrast loss to encourage identity diversity and maintain identity consistency. In order to further improve the recognition accuracy of NIR images, relevant technicians have proposed a heterogeneous face recognition method based on instance-level spatial guidance, which is applied to a pair of spatial adaptive normalization modules and semantic space distribution adaptive alignment modules, and a cross-spectral face hallucination network guided by target domain photos is designed. However, the currently known HFR methods mainly focus on reducing domain differences and have achieved good performance. However, they ignore the changes in facial attributes in real applications, and often cause content changes when generating faces, which cannot well maintain identity characteristics, and thus easily cause the problem of unclear generated details. Summary of the invention

[0005] In view of the above situation, the main purpose of the present invention is to propose a dual-coding face synthesis method and system for infrared-visible light face recognition to solve the above technical problems.

[0006] The present invention proposes a dual-coded face synthesis method for infrared-visible light face recognition, the method comprising the following steps:

[0007] Step 1: Using a dual encoding structure of an identity encoder and a style encoder, taking an infrared face photo as a real photo, inputting the real photo into the identity encoder to obtain the identity feature code;

[0008] Step 2: Use a random visible light face photo as a reference photo, input the reference photo and the real photo into a style encoder, and obtain a content code with the style information of the reference photo;

[0009] Step 3: Fusion-enhance the content coding and identity feature coding to obtain enhanced features;

[0010] Step 4: Input the enhanced features into the decoder, and establish skip connections between the style encoder and the decoder to pass the feature information captured by the style encoder to the style embedded decoder, thereby guiding the generation of decoding features and obtaining the generated results.

[0011] The present invention also proposes a dual-coded face synthesis system for infrared-visible light face recognition, wherein the system applies the dual-coded face synthesis method for infrared-visible light face recognition as described above, and the system comprises:

[0012] Identity information extraction module, used to:

[0013] The dual encoding structure of identity encoder and style encoder is adopted. The infrared face photo is used as the real photo. The real photo is input into the identity encoder to obtain the identity feature code.

[0014] Style-content information extraction module, used to:

[0015] A random visible light face photo is used as a reference photo, and the reference photo and the real photo are input into a style encoder to obtain a content encoding with the style information of the reference photo;

[0016] Feature fusion module, used for:

[0017] The content coding and the identity feature coding are fused and enhanced to obtain enhanced features;

[0018] Image generation module for:

[0019] The enhanced features are input into the style embedding decoder, and skip connections of each layer are established between the style encoder and the style embedding decoder to pass the feature information captured by the style encoder to the style embedding decoder, thereby guiding the generation of decoding features and obtaining the generation results.

[0020] Compared with the prior art, the present invention has the following beneficial effects:

[0021] 1) The method of the present invention utilizes the prior information of the reference photo in the target domain and improves the authenticity of the VIS image generated by NIR image conversion through the fusion of global and local style information, thereby improving the accuracy of NIR face recognition.

[0022] 2) The method of the present invention allows the use of any VIS face image as a reference to guide NIR image conversion, thereby improving the robustness of the model in different face databases and face recognition systems.

[0023] 3) Under the influence of the attention mechanism, the method of the present invention focuses on learning the facial features of the human face, improves the quality of image synthesis, and can make the contours and details of the facial features more prominent and clear.

[0024] 4) The method of the present invention uses a face recognition model as an identity feature extraction model, which enhances the identity retention of face generation and effectively stabilizes the face recognition accuracy.

[0025] 5) The method of the present invention can directly use the existing VIS image recognition for recognition and can be seamlessly embedded in the existing face recognition system.

[0026] 6) The method of the present invention can maintain a high degree of consistency with the source image content when synthesizing a human face. When converting an NIR image to a VIS image, it can restore the unclear details of the eyeball, eyelashes, etc. in the NIR domain.

[0027] 7) The method of the present invention can well distinguish facial contours, facial features, and hair, so that the characteristics of each part can be effectively learned, so that the generated VIS image has different coloring in different parts.

[0028] 8) The dual-branch feature encoding enhances the learning of identity features and increases the diversity of facial attributes. Different reference photos generate VIS photos with different skin colors, which has the effect of data enhancement.

[0029] Additional aspects and advantages of the present invention will be given in part in the following description and in part will be obvious from the following description or learned through embodiments of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] Figure 1 This is a flow chart of the dual-encoded face synthesis method for infrared-visible light face recognition proposed by the present invention;

[0031] Figure 2 This is the overall architecture diagram of the dual-coding face synthesis method for infrared-visible light face recognition proposed by the present invention;

[0032] Figure 3 For the present invention Structure diagram of

[0033] Figure 4 For the present invention Structure diagram of

[0034] Figure 5 This is a network structure diagram of the style embedded decoder of the present invention;

[0035] Figure 6 This is a schematic diagram of the structure of the dual-encoding face synthesis system for infrared-visible light face recognition proposed by the present invention;

[0036] Figure 7 This is a schematic diagram of the overall process of the present invention from data preparation, model construction, training optimization to recognition. DETAILED DESCRIPTION

[0037] Embodiments of the present invention are described in detail below, examples of which are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present invention, and cannot be understood as limiting the present invention.

[0038] These and other aspects of the embodiments of the present invention will be apparent with reference to the following description and accompanying drawings. In these descriptions and accompanying drawings, some specific implementations of the embodiments of the present invention are specifically disclosed to represent some ways of implementing the principles of the embodiments of the present invention, but it should be understood that the scope of the embodiments of the present invention is not limited thereto.

[0039] See also Figure 1 This embodiment provides a dual-coded face synthesis method for infrared-visible light face recognition, the method comprising the following steps:

[0040] Step 1: Using a dual encoding structure of an identity encoder and a style encoder, taking an infrared face photo as a real photo, inputting the real photo into the identity encoder to obtain the identity feature code;

[0041] In step 1, an infrared face photo is taken As a real photo input identity encoder, the process of obtaining identity feature encoding has the following relationship:

[0042] ;

[0043] ;

[0044] ;

[0045] in, express The convolution of represents the maximum pooling, represents the output feature of the i-th layer, represents average pooling, represents the residual block layer, represents the output features of the identity encoder, It indicates that the i-th layer is a feature extraction module consisting of a convolutional layer and a fully connected layer, without changing the number of channels of the feature map.

[0046] In this embodiment, the identity encoder uses the network architecture of the pre-trained model LightCNN_29Layers_v2, which consists of five layers of downsampling. The input image vector x is convolved and pooled to adjust the number of channels to obtain the feature map , where i = 1, 2, 3, 4, 5. Input a real picture, and after five layers of downsampling, the output size is The feature map of

[0047] In order to effectively integrate with the identity encoding, the style encoder is set with five downsampling layers. After convolution, normalization, and activation, the feature map is obtained. , where i=1, 2, 3, 4, 5, and the final output feature map.

[0048] like Figure 2 As shown, it can be seen that the style encoder It consists of five convolutional blocks and 1 Each convolution block consists of a and a Composition, starting from the second convolution block, the feature size is reduced by 2 times, five convolution blocks Respectively expressed as:

[0049] ;

[0050] ;

[0051] ;

[0052] ;

[0053] ;

[0054] It consists of 9 residual adain layers, each of which consists of an original adain module and a residual block.

[0055] Step 2: Use a random visible light face photo as a reference photo, input the reference photo and the real photo into a style encoder, and obtain a content code with the style information of the reference photo;

[0056] In step 2, the visible light face photo and the infrared face photo are input into a style encoder, and the method for obtaining content encoding with style information of the visible light face photo specifically includes the following steps:

[0057] The style encoder inputs both real photos and reference photos, extracting the content information of different network layers of the face and the style information of the reference photo respectively. The corresponding process relationship is as follows:

[0058] ;

[0059] ;

[0060] ;

[0061] ;

[0062] in, represents the feature vector of the real photo at the i-th layer, represents the feature vector of the i-th layer reference photo, represents the i-th convolutional layer, A visible light photograph representing a reference;

[0063] The process relationship of fusing the feature vector of the fifth-level real photo and the feature vector of the reference photo to obtain the global fusion feature is as follows:

[0064] ;

[0065] in, It means that the fifth layer combines the fusion features of content and style information. represents the global style transfer module used to fuse content and global style information, Represents the output features of the style encoder;

[0066] like Figure 3 As shown in Figure 1, the global style transfer module consists of 9 style transfer layers, each of which consists of an original style transfer module and a residual block. The residual block consists of two 3×3 convolution blocks and a residual connection. The feature vector of the real photo and the feature vector of the reference photo are fused through the global style transfer module to achieve style transfer and obtain the fused feature C. The fusion process of the global style transfer module can be expressed as:

[0067] ;

[0068] in, represents a convolution block that does not change the vector size, represents the j-th style transfer layer, represents the features after fusion of the j-th style transfer layer, represents the residual block, Represents the style transfer module. The calculation process of the style transfer module has the following relationship:

[0069] ;

[0070] in, represents the variance of the eigenvector, Represents the mean of the feature vector, and the real face content is represented by To scale, use to transfer;

[0071] In this step, in order to fuse the global style features, the present invention designs a global style transfer module in the fifth layer of the style encoder, which consists of a style transfer module and a residual block, wherein the style transfer module can receive a content input x and a style input y, and align the channel mean and variance of x to match the mean of y, thereby transferring the style information to the source domain.

[0072] Step 3: Fusion-enhance the content coding and identity feature coding to obtain enhanced features;

[0073] In this step, the content with style features is encoded and identity feature encoding obtained by pre-training model Input Vit module to enhance feature correlation, obtain enhanced features of the image through encoding, and associate the two features to obtain a whole , the whole process can be defined as:

[0074] ;

[0075] in, represents the enhanced features, Represents the content encoding that incorporates global style information, Indicates identity feature code, Represents the pixel-wise ViT network module, Represents the concatenation function.

[0076] Step 4: Input the enhanced features into the decoder, and establish jump connections between the style encoder and the style embedding decoder at each layer to pass the feature information captured by the style encoder to the style embedding decoder, thereby guiding the generation of decoding features and obtaining the generation results.

[0077] The network structure of the decoder in this embodiment is as follows Figure 4 As shown in Figure 1, the corresponding style encoder constructs five deconvolution layers. The enhanced features are stretched through the fully connected layer to adjust the feature vector to a size of 128×4×4.

[0078] The stretched vector is then upsampled through the deconvolution layer and fused with the features of the reference photo through the local style transfer module to obtain the upsampled fusion feature. The fifth layer of upsampled fusion features is then passed through a convolution block to obtain the output vector.

[0079] In step 4, the enhanced features are input into the decoder, and a skip connection of each layer is established between the style encoder and the decoder to pass the feature information captured by the identity encoder to the decoder, thereby guiding the generation of the decoding features. The method for obtaining the generation result specifically includes the following steps:

[0080] The enhanced feature f is stretched through the fully connected layer FC to adjust it to a stretched vector of size 128×4×4 so that it can be subsequently upsampled by a factor of 2.

[0081] ;

[0082] Among them, FC represents the fully connected layer, Represents the stretched vector.

[0083] In order to ensure the authenticity of the style, the enhanced features are input into the decoder After that, a deconvolution operation is performed, and then the style feature vector of the fifth layer reference photo in the style encoder is locally fused to obtain the first layer upsampled fusion feature. The corresponding process relationship is as follows:

[0084] ;

[0085] ;

[0086] in, represents the deconvolution layer, represents the local style transfer module used to fuse content and local style information, represents the i-th layer deconvolution vector, represents the upsampled fusion features of the i-th layer;

[0087] Among them, the deconvolution layer consists of residual blocks and deconvolution blocks, and its structure can be expressed as:

[0088] ;

[0089] ;

[0090] in, represents a deconvolution block that increases the vector size by a factor of 2. Represents a 3×3 convolutional block that does not change size.

[0091] In order to keep the conversion content unchanged, the jump connection is used to add the first layer of upsampled fusion features to the feature vector of the fifth layer of real photos in the style encoder and then perform deconvolution. Then, it is locally fused with the style feature vector of the fourth layer of reference photos in the style encoder to generate the second layer of upsampled fusion features. And so on. The corresponding process relationship is as follows:

[0092] ;

[0093] ;

[0094] The upsampled fusion features of the fifth level are convolved to obtain the mask operation and combined with the real image to obtain the output image.

[0095] Preferably, the local style transfer module consists of two style modules, each of which consists of a style attention module and a convolutional layer, and the corresponding process has the following relationship:

[0096] ;

[0097] in, Indicates that the i-th layer deconvolution vector passes through The fusion characteristics of represents the first style module, Represents the second style module, Represents the style attention module;

[0098] like Figure 5 As shown, the style attention module consists of 3 It consists of a convolutional layer and an attention layer, which can easily fuse style features and enrich global style and local style statistics. The corresponding relationship is:

[0099] ;

[0100] ;

[0101] in, Respectively and The features after convolution, represents the attention layer.

[0102] Preferably, the method of convolving the upsampled fusion features of the fifth level to obtain a mask operation and combining the real image to obtain an output image specifically includes the following steps:

[0103] The upsampled fusion features of the fifth level are added to the feature vectors of the first level real photos and then convolved to obtain the final output features. The corresponding process relationship is as follows:

[0104] ;

[0105] in, represents the convolutional residual block, represents the up-sampled fusion features of the fifth layer, and F represents the final output features;

[0106] The convolutional residual block consists of a convolutional block and a residual block, which can be expressed as:

[0107] ;

[0108] The final output feature is subjected to a mask operation to obtain a content mask and an attention mask. Combined with the real photo, the final result is generated based on the content mask and the attention mask. The corresponding process relationship is as follows;

[0109] ;

[0110] in, Indicates the content mask, Indicates attention mask, represents the output image obtained by fusing the content mask, attention mask and input image.

[0111] The process of obtaining the attention mask has the following relationship:

[0112] ;

[0113] in, represents the activation function, represents the dimension expansion function, It means taking the first column data of all rows in the output feature;

[0114] The process of obtaining the content mask has the following relationship:

[0115] ;

[0116] It means taking the data of all columns except the first column of all rows in the output feature.

[0117] In this step, a module for acquiring attention mask and content mask is built into the decoder, which can perceive the most distinguishing content between the source domain and the target domain, and focus on learning the facial features of the face. The attention mask defines the intensity of each pixel and specifies the degree of contribution of each pixel of the content mask in the final rendered image. In this way, the generator does not need to render background elements and can only focus on the pixels where the target domain content moves, thereby generating a clearer and more realistic synthetic image. In addition, the network structure of the decoder is also set with five layers so as to establish jump connections between the encoder and the decoder to further improve the image clarity and style authenticity. The present invention designs a local style transfer module between the decoder layers, which can fuse the local style of the reference photo to enhance the style rendering. After deconvolution, normalization, activation and fusion, the final output features are obtained.

[0118] Please refer to Figure 6 This embodiment further provides a dual-coded face synthesis system for infrared-visible light face recognition, wherein the system applies the dual-coded face synthesis method for infrared-visible light face recognition as described above, and the system includes:

[0119] Identity information extraction module, used to:

[0120] The dual encoding structure of identity encoder and style encoder is adopted. The infrared face photo is used as the real photo. The real photo is input into the identity encoder to obtain the identity feature code.

[0121] Style-content information extraction module, used to:

[0122] A random visible light face photo is used as a reference photo, and the reference photo and the real photo are input into a style encoder to obtain a content encoding with the style information of the reference photo;

[0123] Feature fusion module, used for:

[0124] The content coding and the identity feature coding are fused and enhanced to obtain enhanced features;

[0125] Image generation module for:

[0126] The enhanced features are input into the decoder, and skip connections of each layer are established between the style encoder and the decoder to pass the feature information captured by the identity encoder to the decoder, thereby guiding the generation of decoding features and obtaining the generated results.

[0127] Figure 7 The process of the present invention from data preparation, model construction, training optimization to final recognition is shown in the figure. The specific process is as follows:

[0128] 1) Data preparation

[0129] Download the CASIA NIR-VIS 2.0 and LAMP_HQ heterogeneous face datasets, and divide them into training sets and test sets according to the 10-fold training setting.

[0130] For LAMP_HQ, create a list of names and labels according to the training set and test set, and cut them to size.

[0131] For CASIA NIR-VIS 2.0, the positioning function in the face_recognition library is used to locate the rectangular area of ​​the face, extract the regional photo and save it, and then use the pre-trained face feature point detection model shape_predictor_68_face_landmarks to automatically detect the 68 key points of the face, and align the face according to the key points. The specific operation is to calculate the angle between the line connecting the two eyes and the horizontal line, and then obtain the corresponding rotation matrix through the angle. The aligned face is scaled to a size of Similarly, make a list of names and labels for the training and test sets.

[0132] Before injecting the dataset into the model, the images are augmented by center rotation and horizontal flipping, and then converted into vectors and normalized with a three-channel mean of 0.5 and a variance of 0.5.

[0133] 2) Model construction

[0134] Cycle consistency loss Indicates the fake image generated by the generator after the input real photo Then through the generator React Map's restore image requires Consistent with the input image, identity consistency loss Indicates that the target domain photo The converted image of the generator as the source domain photo input must be consistent with itself. They are expressed as:

[0135] The model designed by the method of the present invention is based on CycleGAN, which includes two pairs of generative adversarial networks, which are responsible for modeling two mapping directions: the direction of conversion from NIR (near infrared) to VIS (visible light), and the generator Take the NIR image as input and the VIS image as reference image to guide the generation of NIR (VIS image corresponding to near infrared; the direction of conversion from VIS to NIR, the generator The VIS image is used as input and the NIR image is used as a reference image to guide the generation of the NIR image corresponding to the VIS. To explain, the generator Similar, such as Figure 2 As shown, the generator It consists of three parts, including identity-style dual encoder, feature enhancement module, and style embedding decoder.

[0136] 3) Parameter Tuning

[0137] 3.1) Fighting against loss

[0138] Adversarial Loss Function Is a generator With the discriminator The mutual competition between the two networks can improve the ability of the generative network to generate real photos and the ability of the discriminator to distinguish between real photos and fake photos.

[0139] For the generator , loss function It can be expressed as:

[0140] ;

[0141] in, Identify whether the generated vis image is real. Represents the generated fake vis picture, Represents the mathematical expectation of the distribution of generated images.

[0142] For the discriminator , the loss function can be expressed as:

[0143] ;

[0144] in, Indicates the VIS domain and Matching real photos.

[0145] In summary, for the generative adversarial network from NIR to VIS conversion, the adversarial loss function It can be expressed as:

[0146] ;

[0147] 3.2) Cycle consistency loss The real photo input passes through the generator Generated fake images Then through the generator The reverse mapping of the restored image requires Consistent with the input image, identity consistency loss Indicates that the target domain photo The converted image of the generator as the source domain photo input must be consistent with itself. Cycle consistency loss and identity consistency loss Respectively expressed as:

[0148] ;

[0149] ;

[0150] 3.3) Identity feature loss

[0151] In order to enhance the identity preservation between the input image and the generated image, the content preservation loss is introduced , so that the input image and the output image content are consistent, by calculating the VGG-19 deep convolutional neural network extracted and The Euclidean distance between features gives the content loss, which can be expressed as:

[0152] ;

[0153] in, represents the square of the Euclidean distance, It represents the operation of obtaining the j-th layer output feature of the VGG-19 deep convolutional neural network. The method of the present invention uses the conv_4 layer to extract content features. Respectively represent the number of channels, height and width of the feature vector.

[0154] In order to increase the details of the generated image, matching loss is added to enable more subtle imaging of facial features such as eyes and mouth. , let the output image keep the identity features consistent with the matching image, calculate the pixel-level L1 loss (absolute error loss) of the image, and take the features before the last FC layer of LightCNN_29Layers_v2 for L1 loss calculation, which can be expressed as:

[0155] ;

[0156] in, represents the matching loss, represents the absolute error, Represents the LightCNN_29Layers_v2 model with the last FC layer removed.

[0157] 3.4) Style Feature Loss

[0158] The style features brought by the reference information are not enough to make the face style completely consistent with the target domain. In order to make the style more realistic, style loss is added. , which is calculated by and The Gram matrix distance between them can be expressed as:

[0159] ;

[0160] in, Represents the Gram matrix.

[0161] ;

[0162] in, The Gram matrix representing the features of X extracted by model U at layer j, represents the features of X extracted by model U at layer j, Respectively represent the height and width of the feature vector, represents two different channels of the feature vector, Representing the input image, the method of the present invention uses conv_1, conv_2, conv_3, conv_4 and conv_5 layers to extract style features.

[0163] In summary, total loss By integrating the above losses we get:

[0164] ;

[0165] in, Indicates complete loss. In this embodiment, , in order to coordinate the various loss functions to achieve the best effect of generating images.

[0166] 4) Face recognition

[0167] After the model training is completed, the generator for converting NIR images to VIS images is taken The performance of face recognition is evaluated using the CASIA NIR-VIS 2.0 and LAMP_HQ test sets. Taking the LAMP_HQ test set as an example, the face library Gallery contains VIS images of N people. , each identity has only one photo, that is: ;

[0168] For any NIR image to be identified in the test set ,Will As the matching image, Use as a reference image to guide NIR images Generate VIS images corresponding to identity and style ,in and Represents two VIS face images of different identities, namely: ;

[0169] Use the pre-trained LightCNN29 model as the face recognition model , respectively extract and Identity characteristics and , calculate the cosine similarity , traverse each VIS image in the face library , the final recognition result is determined by locating the maximum cosine similarity, that is:

[0170] ;

[0171] in, The corresponding identity is The recognition result.

[0172] It should be understood that the various parts of the present invention can be implemented by hardware, software, firmware or a combination thereof. In the above-mentioned embodiments, multiple steps or methods can be implemented by software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented by hardware, as in another embodiment, it can be implemented by any one of the following technologies known in the art or their combination: a discrete logic circuit having a logic gate circuit for implementing a logic function for a data signal, a dedicated integrated circuit having a suitable combination of logic gate circuits, a programmable gate array (PGA), a field programmable gate array (FPGA), etc.

[0173] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representation of the above terms does not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described can be combined in any one or more embodiments or examples in a suitable manner.

[0174] The above-mentioned embodiments only express several implementation methods of the present invention, and the description thereof is relatively specific and detailed, but it cannot be understood as limiting the scope of the patent of the present invention. It should be pointed out that, for ordinary technicians in this field, several variations and improvements can be made without departing from the concept of the present invention, which all belong to the protection scope of the present invention. Therefore, the protection scope of the patent of the present invention shall be subject to the attached claims.

Claims

1. A dual-coded face synthesis method for infrared-visible light face recognition, characterized in that: The method comprises the following steps: Step 1: Using a dual encoding structure of an identity encoder and a style encoder, taking an infrared face photo as a real photo, inputting the real photo into the identity encoder to obtain the identity feature code; Step 2: Use a random visible light face photo as a reference photo, input the reference photo and the real photo into a style encoder, and obtain a content code with the style information of the reference photo; Step 3: Fusion-enhance the content coding and identity feature coding to obtain enhanced features; Step 4: Input the enhanced features into the style embedding decoder, and establish skip connections at each layer between the style encoder and the style embedding decoder to pass the feature information captured by the style encoder to the style embedding decoder, thereby guiding the generation of decoding features and obtaining the generated results; In step 4, the enhanced features are input into the style embedding decoder, and skip connections of each layer are established between the style encoder and the style embedding decoder to pass the feature information captured by the style encoder to the style embedding decoder, thereby guiding the generation of decoding features. The method for obtaining the generation result specifically includes the following steps: The enhanced features are stretched through the fully connected layer to obtain a stretched vector for subsequent upsampling. The corresponding process has the following relationship: ; in, represents the fully connected layer, represents the stretched vector; Input the adjusted features into the decoder After that, a deconvolution operation is performed, and then the style feature vector of the fifth layer reference photo in the style encoder is locally fused to obtain the first layer upsampled fusion feature. The corresponding process relationship is as follows: ; ; in, represents the deconvolution layer, represents the local style transfer module used to fuse content and local style information, represents the i-th layer deconvolution vector, represents the upsampled fusion features of the i-th layer; Using skip connections, the first layer of upsampled fusion features are added to the content feature vector of the fifth layer of real photos in the style encoder and then deconvolved, and then fused with the style feature vector of the fourth layer of reference photos to generate the second layer of upsampled fusion features, and so on, until the fifth layer of upsampled fusion features are obtained. The corresponding process relationship is as follows: ; ; Convolve the upsampled fusion features of the fifth level to obtain a mask operation, and combine it with the real image to obtain the output image; The local style transfer module consists of two style modules, each of which consists of a style attention module and a convolutional layer. The corresponding process has the following relationship: ; ; ; ; ; in, represents the fusion feature of the i-th layer deconvolution vector after the first style module, represents the first style module, Represents the second style module, Represents the style attention module; The method of convolving the upsampled fusion features of the fifth level to obtain a mask operation and combining the real image to obtain an output image specifically includes the following steps: The upsampled fusion features of the fifth level are added to the feature vectors of the first level real photos and then convolved to obtain the final output features. The corresponding process relationship is as follows: ; in, represents the convolutional residual block, represents the up-sampled fusion features of the fifth layer, and F represents the output features after convolution; The output features are subjected to masking operations to obtain content masks and attention masks, and the final results are generated based on the content masks and attention masks. The corresponding process relationship is as follows: ; in, Indicates the content mask, Indicates attention mask, represents the fusion of content mask, attention mask and input image to obtain output image; The process of obtaining the attention mask has the following relationship: ; in, represents the activation function, represents the dimension expansion function, It means taking the first column data of all rows in the output vector; The process of obtaining the content mask has the following relationship: ; It means to take the data of all columns except the first column of all rows in the output vector.

2. The dual-coded face synthesis method for infrared-visible light face recognition according to claim 1, characterized in that: In step 1, the identity encoder adopts the network architecture of the pre-trained model LightCNN_29Layers_v2, and the network architecture consists of five layers of downsampling; The style encoder is configured with five downsampling layers.

3. The dual-coded face synthesis method for infrared-visible light face recognition according to claim 2, characterized in that: In step 1, an infrared face photo is taken As a real photo input identity encoder, the process of obtaining identity feature encoding has the following relationship: ; ; ; in, express The convolution of represents the maximum pooling, represents the output feature of the i-th layer, represents average pooling, represents the i-th residual block layer, represents the output features of the identity encoder, It indicates that the i-th layer is a feature extraction module consisting of a convolutional layer and a fully connected layer.

4. The dual-coded face synthesis method for infrared-visible light face recognition according to claim 3, characterized in that: In step 2, the visible light face photo and the infrared face photo are input into a style encoder, and the method for obtaining content encoding with style information of the visible light face photo specifically includes the following steps: The style encoder inputs both real photos and reference photos, extracting the content information of different network layers of the face and the style information of the reference photo respectively. The corresponding process relationship is as follows: ; ; ; ; in, represents the content features of the i-th layer of real photos, represents the style features of the i-th layer reference photo, represents the i-th convolutional layer, A visible light photograph representing a reference; The feature vector of the fifth-level real photo and the feature vector of the reference photo are fused to obtain the global fusion feature. The corresponding process relationship is as follows: ; ; in, It means that the fifth layer combines the fusion features of content and style information. represents the global style transfer module used to fuse content and global style information, Represents the output features of the style encoder.

5. The dual-coded face synthesis method for infrared-visible light face recognition according to claim 4, characterized in that: The global style transfer module consists of 9 style transfer layers, each of which consists of an original style transfer module and a residual block; The residual block consists of two 3×3 convolutional blocks and a residual connection; The feature vector of the real photo and the feature vector of the reference photo are fused through the global style transfer module to achieve style transfer and obtain the fused feature C; The fusion process of the global style transfer module can be expressed as: ; ; ; ; ; ; in, represents a convolution block that does not change the vector size, represents the j-th style transfer layer, represents the features after fusion of the j-th style transfer layer, represents the residual block, Represents the style transfer module.

6. The dual-coded face synthesis method for infrared-visible light face recognition according to claim 5, characterized in that: In step 3, the content coding and identity feature coding are fused and enhanced, and the process of obtaining the enhanced features has the following relationship: ; in, represents the enhanced features, Represents the content encoding that incorporates global style information, Indicates identity feature code, Represents the pixel-wise ViT network module, Represents the concatenation function.

7. A dual-coding face synthesis system for infrared-visible light face recognition, characterized in that: The system applies the dual-coding face synthesis method for infrared-visible light face recognition according to any one of claims 1 to 6, and the system comprises: Identity information extraction module, used to: The dual encoding structure of identity encoder and style encoder is adopted. The infrared face photo is used as the real photo. The real photo is input into the identity encoder to obtain the identity feature code. Style-content information extraction module, used to: A random visible light face photo is used as a reference photo, and the reference photo and the real photo are input into a style encoder to obtain a content encoding with the style information of the reference photo; Feature fusion module, used for: The content coding and the identity feature coding are fused and enhanced to obtain enhanced features; Image generation module for: The enhanced features are input into the style embedding decoder, and skip connections of each layer are established between the style encoder and the style embedding decoder to pass the feature information captured by the identity encoder to the decoder, thereby guiding the generation of decoding features and obtaining the generated results.

Citation Information

Patent Citations

  • Human face visible light image generation method and related device

    CN112258592A

  • Multi-deformation style face cartoon automatic generation method based on feature map partitioning

    CN112837210A

  • Infrared mask face image generation method and device and face recognition system

    CN114926458A

  • Three-dimensional face reconstruction method based on single image

    CN117036620A

  • Multi-modal guided face image editing method

    CN117830474A