A method for converting thermal infrared to visible light images of faces based on prior information

Abstract

The present invention belongs to the field of computer vision and artificial intelligence, and specifically relates to a method for converting thermal infrared-visible light images of faces based on prior information. Under the contrastive learning framework, the present invention designs a method based on a face parsing graph as prior information to guide the generation network to learn the local texture information of face images. The thermal infrared-visible light generation network model of the face based on prior information mainly includes a face parsing graph conditional network module, a spatial feature transformation mapping layer, an attention module, a generator network module and a discriminator; the model is converted through the spatial feature transformation mapping layer STL, which uses the face parsing graph mapping features as a priori conditions to generate a pair of modulation parameters, and performs affine transformation on the face features of the generation network according to the modulation parameters, thereby adaptively optimizing the generation quality of the face image, and at the same time supervises the learning together with the designed face gradient enhancement loss, which is conducive to alleviating the appearance of artifacts on the face generated image, improving local texture details, and making the image generation restore the corresponding face attribute information as much as possible.

Description

Technical Field

[0001] The present invention belongs to the field of computer vision and artificial intelligence, and specifically relates to a method for converting thermal infrared-to-visible light images of human faces based on prior information. Background Art

[0002] With the increasing prevalence of various sensing instruments in daily life, image data from various modalities has become increasingly abundant. In the field of computer vision, transforming and learning mapping relationships between images of different modalities is a key research direction. Thermal infrared image sensors, among other things, are a novel imaging method that captures information describing both the target and the scene based on the radiant energy between them. Therefore, they are suitable for security surveillance and military reconnaissance applications. However, unlike near-infrared images, thermal infrared facial images suffer from poor visibility, low contrast, a lack of detailed texture information and details, and high sensitivity to temperature differences, making them difficult to effectively recognize and analyze. In contrast, visible light facial images possess rich texture details, high contrast, and high visibility, and numerous mature visible light face recognition and analysis methods have been developed. Therefore, converting thermal infrared facial images into visible light facial images can effectively recognize and analyze facial information in thermal infrared images.

[0003] Numerous cross-modal generative frameworks have been developed to address the conversion between thermal infrared and visible light facial images. General conversion frameworks fall into two main categories: generative adversarial networks (GANs) and their variants, and variational autoencoders (VAEs) and their variants. Most current visible-light-to-thermal infrared facial image conversion algorithms are based on GANs. For example, pix2pix, a general image conversion framework based on a U-Net architecture, uses an L1 loss to measure the difference between generated and real images. These methods require matched thermal infrared-to-visible light facial data. Consequently, these methods suffer from insufficient and difficult-to-obtain pixel-level paired datasets, making them inadequate for training high-quality face synthesis models. CycleGAN, a classic unpaired image conversion framework, enables bidirectional image conversion and introduces a cycle consistency loss, which facilitates the generation of image details to a certain extent. However, the synthesized facial images still exhibit distortion and artifacts.

[0004] In summary, when using the thermal infrared-visible light face generation method, the generated visible light face images may have facial attribute issues, such as skin color issues, generated image artifacts, and loss of texture details. This significantly affects the quality and visual quality of the generated images. Summary of the Invention

[0005] To solve the problems existing in the above-mentioned prior art, the present invention proposes a method for converting facial thermal infrared to visible light images based on prior information, which is characterized by comprising: obtaining a facial infrared image to be converted and a corresponding visible light paired image, inputting the facial infrared image into a trained facial thermal infrared to visible light generative network model based on prior information, and obtaining a facial visible light composite image; the facial thermal infrared to visible light generative network model based on prior information includes a facial parsing graph conditional network module, a spatial feature transformation mapping layer, an attention module, a generator network module, and a discriminator;

[0006] The process of training the face thermal infrared-visible light generation network model based on prior information includes:

[0007] S1: Obtain a thermal infrared-visible light image dataset and classify the skin color information labels of the images in the dataset;

[0008] S2: Preprocess the paired images in the dataset and input the preprocessed images into the face thermal infrared-visible light generation network model based on prior information;

[0009] S3: Use the face parsing graph conditional network module to extract features from the preprocessed face parsing graph to obtain the face prior information features;

[0010] S4: Using the generator network module to extract and encode the preprocessed facial thermal infrared image to obtain encoded facial feature information;

[0011] S5: Inputting the facial prior information features and the encoded facial feature information into the spatial feature transformation mapping layer to generate a pair of modulation parameters; mapping the encoded facial feature information according to the modulation parameters to obtain mapped facial feature information;

[0012] S6: The facial feature information undergoes multiple layers of residual transformation and decoder to obtain the corresponding visible light synthetic face image, which is then input into the discriminator for discrimination training.

[0013] S7: Input the visible light composite face image and the thermal infrared face image into the encoder and two MLP mapping layers to obtain the corresponding thermal infrared face image features and visible light composite face image features; input the thermal infrared face image features and the visible light composite face image features into the attention module for contrastive learning, and calculate the model loss function according to the contrastive learning formula;

[0014] S8: The Adam optimizer is used to optimize the model parameters. When the loss function of the model is minimized, the optimal parameters are output to obtain the optimal face thermal infrared-visible light generation network model based on prior information.

[0015] Preferably, the generator network module includes an encoder G enc , converter and decoder G dec ; Encoder G enc It is mainly composed of 3 layers of CIR. Each layer of CIR is composed of a convolution, InstanceNorm normalization, and ReLU activation function. The encoder extracts features from the input image. The converter is composed of 9 Residual blocks. Each Residual block is composed of a spatial feature transformation mapping layer STL and a CIR layer. This residual module mainly enhances the feature map extracted by the encoder. The decoder G dec It includes two CTIR layers, a Reflect operation layer, and a convolution layer. The CTIR layer is composed of deconvolution, InstanceNorm normalization, and ReLU activation function in sequence; the function of the decoder is upsampling operation, which gradually reconstructs the learned facial features to the original image size.

[0016] Preferably, a spatial feature transformation mapping layer is used to process two input face features. The two inputs are the face prior information features generated by the face parsing condition module and the feature output G of each layer of the generative network. F , where the facial prior information features are respectively subjected to two layers of convolution operations to obtain a pair of parameters α and β, and the facial feature output G in the generative network is modulated according to the parameters F First, do the dot multiplication, then do the addition operation to get the output of the entire STL network. This network can spatially map the facial feature information in the generative network to the corresponding transformation, thereby adaptively optimizing the generated quality of the facial image.

[0017] Preferably, an attention module is used to perform comparative learning on the thermal infrared input image of the face and the synthetic image of the face under visible light, and the process includes:

[0018] S71: Perform multi-layer feature extraction on the thermal infrared image features and the visible light composite image features of the face; that is, the thermal infrared image features and the visible light composite image features of the face are each passed through an encoder G enc and a two-layer MLP network layer H l Get the facial feature spectrum F H ∈R C×H×W and F V ∈R C×H×W ;

[0019] S72: Reshape and transpose the characteristic spectrum of the face thermal infrared image to obtain a two-dimensional matrix Q H ∈R HW×C and V H ∈R HW×C;

[0020] S73: According to the two-dimensional matrix Q H ∈R HW×C and V H ∈R HW×C Constructing a global attention contrastive loss.

[0021] Furthermore, the model has two attention methods: global attention and local attention to construct contrastive learning loss. In this embodiment, global attention is used to construct contrastive learning loss. The process of constructing global attention contrastive loss includes: H Multiply by its transpose K H ∈R C×HW Get a matrix, perform Softmax normalization on each row of the matrix to get a global attention matrix A global ∈R HW×HW ; Calculate the entropy value H of each row in the global attention matrix according to the entropy calculation formula s And sort each row of data in the global attention matrix in ascending order according to the calculated entropy value; according to the sorted matrix, route the V in the source domain face thermal infrared image and the target domain face visible light synthetic image respectively. H ∈R HW×C and V V ∈R HW×C Features; finally route the corresponding value feature V H and V V To construct the global contrast loss.

[0022] Furthermore, the process of constructing the local attention contrast loss includes: the local attention here uses a k×k constant size window to perform a sliding query with a step size of 1 on the source domain face thermal infrared image, which can enhance the spatial information interaction in the local area. H Multiply by its local transpose Get a matrix, perform Softmax normalization on each row of the matrix to get a local attention matrix Calculate A local The entropy value H of each row in s , and arrange each row of data in the local attention matrix in ascending order according to the calculated entropy value, and route the value of the source domain face thermal infrared image according to the arranged matrix and the value of the target domain face visible light synthetic image Thus, a multi-layer local contrast loss is constructed.

[0023] Preferably, the loss function of the model is:

[0024] L=λ1L ConH +λ2L Con G(H) +λ3L Pcl +λ4L Gm +λ5L gan

[0025] Among them, λ1, λ2, λ3, λ4, and λ5 are the hyperparameters of contrastive learning loss, identity-preserving contrastive learning loss, gradient enhancement loss, pixel-level consistency loss, and generative adversarial loss, respectively.

[0026] Beneficial effects of the present invention:

[0027] This invention uses a facial parsing graph as prior information to guide a generative network in learning local texture information of facial images. Using the facial parsing graph mapping features as prior conditions, the spatial feature transform mapping layer (STL) generates a pair of modulation parameters. Based on these modulation parameters, an affine transformation is performed on the facial features of the generative network, thereby adaptively optimizing the generated facial image quality. This helps mitigate artifacts in the generated facial image and improve local texture detail. During model training, the invention designs a facial gradient enhancement loss that extracts the gradient maps corresponding to the visible light composite image and the ground truth (GT) image of the face. This gradient enhancement loss can enhance facial details in the visible light composite image and maintain better facial contour information. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] Figure 1 This is a flow chart of the method for synthesizing thermal infrared and visible light images of a human face according to the present invention;

[0029] Figure 2 This is a schematic diagram of the framework structure of the method for synthesizing thermal infrared and visible light images of a human face according to the present invention;

[0030] Figure 3 Schematic diagram of the spatial feature transformation mapping layer method of the present invention;

[0031] Figure 4 The present invention is a schematic diagram of a face parsing graph conditional network module method;

[0032] Figure 5 This is a schematic diagram of the attention module method of the present invention. DETAILED DESCRIPTION

[0033] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0034] A method for converting thermal infrared-visible light images of human faces based on prior information, such as Figure 1 and Figure 2 As shown, the method includes: first, preparing a paired face thermal infrared-visible light dataset and a corresponding face prior dataset (face parsing graph), and performing skin color information classification and label extraction, as well as corresponding data preprocessing; constructing a face thermal infrared-visible light generation network model based on prior information and a model including a face parsing graph conditional network (FPCN), a spatial feature transformation mapping layer (STL), a generator network module G, a discriminator D, etc.; training the face generation network model, performing corresponding attention operations on the source domain and target domain images, and obtaining significant anchor points and positive and negative samples after selection; combining multiple loss functions and Adam optimizer for training and optimization, and updating network parameters; obtaining the optimal model after training, and inputting the face thermal infrared image into the optimal generation model to obtain a face visible light synthetic image.

[0035] A specific implementation method of a method for converting a facial thermal infrared image into a visible light image based on prior information, the method comprising: obtaining a facial infrared image to be converted and a corresponding visible light paired image, inputting the facial infrared image into a trained facial thermal infrared-visible light generation network model based on prior information, and obtaining a facial visible light composite image; the facial thermal infrared-visible light generation network model based on prior information comprises a facial parsing graph conditional network module, a spatial feature transformation mapping layer, an attention module, a generator network module, and a discriminator.

[0036] A process for training a face thermal infrared-visible light generation network model based on prior information includes:

[0037] S1: Obtain a thermal infrared-visible light image dataset and classify the skin color information labels of the images in the dataset;

[0038] S2: Preprocess the paired images in the dataset and input the preprocessed images into the face thermal infrared-visible light generation network model based on prior information;

[0039] S3: Use the face parsing graph conditional network module to extract features from the preprocessed face parsing graph to obtain the face prior information features;

[0040] S4: Using the generator network module to extract and encode the preprocessed facial thermal infrared image to obtain encoded facial feature information;

[0041] S5: Inputting the facial prior information features and the encoded facial feature information into the spatial feature transformation mapping layer to generate a pair of modulation parameters; mapping the encoded facial feature information according to the modulation parameters to obtain mapped facial feature information;

[0042] S6: The facial feature information undergoes multiple layers of residual transformation and decoder to obtain the corresponding visible light synthetic face image, which is then input into the discriminator for discrimination training.

[0043] S7: Input the visible light composite face image and the thermal infrared face image into the encoder and two MLP mapping layers to obtain the corresponding thermal infrared face image features and visible light composite face image features; input the thermal infrared face image features and the visible light composite face image features into the attention module for contrastive learning, and calculate the model loss function according to the contrastive learning formula;

[0044] S8: The Adam optimizer is used to optimize the model parameters. When the loss function of the model is minimized, the optimal parameters are output to obtain the optimal face thermal infrared-visible light generation network model based on prior information.

[0045] Another specific implementation method for training a face thermal infrared-visible light generation network model based on prior information is as follows: Figure 1 As shown, including:

[0046] S1: Obtain a thermal infrared-visible light image dataset, set the number of iterations, and classify the skin color information labels of the images in the dataset;

[0047] S2: Preprocess the paired images in the dataset and input the preprocessed images into the face thermal infrared-visible light generation network model based on prior information;

[0048] S3: Use the face parsing graph conditional network module to extract features from the preprocessed face parsing graph to obtain the face prior information features;

[0049] S4: Using the generator network module to extract and encode the preprocessed facial thermal infrared image to obtain encoded facial feature information;

[0050] S5: Inputting the facial prior information features and the encoded facial feature information into the spatial feature transformation mapping layer to generate a pair of modulation parameters; mapping the encoded facial feature information according to the modulation parameters to obtain mapped facial feature information;

[0051] S6: The facial feature information undergoes multiple layers of residual transformation and decoder to obtain the corresponding visible light synthetic face image, which is then input into the discriminator for discrimination training.

[0052] S7: Input the visible light composite face image and the thermal infrared face image into the encoder and two MLP mapping layers to obtain the corresponding thermal infrared face image features and visible light composite face image features; input the thermal infrared face image features and the visible light composite face image features into the attention module for contrastive learning, and calculate the model loss function according to the contrastive learning formula;

[0053] S8: Use the Adam optimizer to optimize the parameters of the model, and back-propagate the optimized parameters, and increase the number of iterations by 1; determine the current number of iterations and compare it with the set number of iterations. If they are equal to the set number of iterations, output the optimal parameters to obtain the optimal facial thermal infrared-visible light generation network model based on prior information, otherwise return to step S3.

[0054] Acquiring an infrared-visible image pairing dataset involves using a dual-mode thermal infrared and visible light camera to capture 256x256 thermal infrared and visible light facial datasets. Face alignment is then performed, and the RetinaFace face detection algorithm is used to locate the five key points of the face. The face is then cropped to obtain the corresponding thermal infrared-visible light facial dataset. Skin color information is classified and corresponding labels are extracted from the cropped and uncropped thermal image-visible light facial datasets for subsequent model training. Furthermore, the corresponding facial input prior information (the facial parsing graph dataset) is generated using the latest facial parsing graph synthesis model.

[0055] In this embodiment, if Figure 4 As shown in the figure, the Face Parsing Graph Conditional Network (FPCN) module: This conditional network takes the face parsing graph as input and processes it through three convolutional layers. The convolutional layers here use 1×1 and 3×3 convolution kernels to extract facial features.

[0056] Spatial feature transformation mapping layer (STL): such as Figure 3 As shown, this module has two inputs, namely the face prior information features generated by the FPCN module and the feature output G of each layer of the generative network. F The facial prior information features are respectively obtained through multi-layer convolution operations and sigmoid activation functions to obtain a pair of parameters α and β. According to the modulation parameters, the facial feature output G in the generative network is F First, do the dot multiplication, then do the addition operation to get the output of the entire STL network. This network can spatially map the facial feature information in the generative network to the corresponding transformation, thereby adaptively optimizing the generated quality of the facial image.

[0057] Generator network module G: mainly includes encoder G enc , a converter consisting of 9 Residual blocks, a decoder G dec composition.

[0058] In this embodiment, the encoder G enc It mainly consists of 3 layers of CIR, each of which is composed of a convolution, InstanceNorm normalization, and ReLU activation function operation. A Reflect operation layer (border filling) is inserted before the 3 layers of CIR. This operation fills the image symmetrically along the edges to increase the image resolution. enc The main function is to extract facial features.

[0059] In this embodiment, the converter consists of multiple residual blocks, primarily comprising nine residual blocks. Each residual block sequentially comprises a spatial feature transformation mapping layer and a CIR layer. This residual module is primarily used for feature enhancement. Using the facial parsing image as prior information, the modulation parameters generated by the spatial feature transformation mapping layer are used to transform the features of the generative network to adapt them to the output, significantly improving the texture details of the generated facial image.

[0060] In this embodiment, the decoder G dec It primarily consists of two CTIR layers, a Reflect layer, and a convolution layer. The CTIR layer, in turn, consists of deconvolution, InstanceNorm normalization, and ReLU activation. The decoder gradually reconstructs the image into the original-size visible-light image of the face.

[0061] In this embodiment, the discriminator uses a PatchGAN architecture. Its main focus is to output an N×N matrix for comparison, primarily considering differences in global receptive field information and maintaining high resolution and detail for image clarity. Each time, it selects a 70×70 image patch from the original image to determine authenticity, ultimately outputting a 30×30 matrix. The mean of this matrix is used as the True / False output.

[0062] like Figure 5 As shown in Figure 2, the process of using the attention module to perform comparative learning on the target image and the input image includes:

[0063] Step 1: Multi-layer feature extraction of source and target domain images: This module passes an encoder G on the source domain image (human face thermal infrared input image) and the target domain image (human face visible light synthetic image). enc and two-layer MLP network layer H l The thermal infrared input image of the face passes through the encoder G encExtract multi-layer feature spectra and select L layers of feature maps after encoding. Each layer of feature has S spatial positions and project its features into a shared embedding space through a 2-layer MLP network. Different layers and different spatial positions represent different image blocks. The relationship after image block mapping is: {y l} L =H l {G e l nc (x)} L , l represents the output feature of the layer, l∈{1,2,3,...,L}. The same applies to the target domain image. After encoding and mapping the target domain visible light synthetic face image, one image block is taken as the anchor point, and the image block at the corresponding position in the source domain thermal infrared face input image is used as the positive sample, while the image blocks at other positions of the same face thermal infrared input image are used as negative samples.

[0064] Step 2: Attention Module in Contrastive Learning: This attention module is primarily designed to address the problem of selecting the locations of positive and negative samples in contrastive learning. Previously, random image blocks were selected for constraint, which may not be appropriate. Because certain image locations contain less salient information from the source domain, only those containing salient information from important areas need to be selected. This method of constructing a contrastive loss is more conducive to ensuring cross-domain consistency. This model uses two attention methods: global attention and local attention to construct contrastive learning losses. In this example, global attention is used to construct the contrastive learning loss.

[0065] Step 2.1: The source domain face thermal infrared input image and the target domain face visible light synthetic image are both passed through an encoder G enc and a two-layer MLP network layer H l Get a feature map F respectively H ∈R C×H×W and F V ∈R C×H×W In this embodiment, the global attention method is used. First, the features of the thermal infrared image of the face are reshaped and transposed to obtain a two-dimensional matrix Q H ∈R HW×C and V H ∈R HW×C , Q H Multiply by its transpose K H ∈R C×HW Get a matrix, and then perform Softmax normalization on each row of the matrix to get a global attention matrix A global ∈R HW ×HWEntropy can be used as an indicator to measure the importance of features, so it can be used according to A global The calculated entropy value H for each row in s To measure the importance of features. Where i and j correspond to A global The index position of the rows and columns in the s The global calculation formula is:

[0066]

[0067] Calculate A global After the entropy value of each row is calculated, sort in ascending order according to the entropy value and select the minimum N rows as the global attention matrix A. global-s ∈R N×HW The matrix is routed to the source domain human face thermal infrared image and the target domain human face visible light synthetic image V H ∈R HW×C and V V ∈R HW×C feature.

[0068] like Figure 5 Shown A global-s It is applied to the features from the thermal infrared input image of the face and the visible light synthetic image of the face, routing the corresponding value features (V H and V V ) to form corresponding anchor points, positive samples, and negative samples. The positive and negative samples are located in the source domain thermal infrared input image of the face, while the anchor points are located in the target domain visible light synthetic image of the face, thereby establishing the corresponding global contrast loss.

[0069] Step 2.2: The difference between the local attention method and the local attention method is that the local attention method uses a k×k constant size window and performs a sliding query with a step size of 1 on the source domain face thermal infrared input image. This can strengthen the spatial information interaction and connection in the local area. First, the feature map of the face thermal infrared image is reshaped and transposed to obtain a two-dimensional matrix Q H ∈R HW×C and V H ∈R HW×C , which is different from global attention: here Q H Multiply by its local transpose Get a matrix, and then perform Softmax normalization on each row of the matrix to get a local attention matrix Also according to A global The entropy value H of each row in s To measure the significance of the feature, and then, like the global attention, select the first N rows in ascending order according to the entropy value to form the selected local attention matrix A local-s, and then route the values of the source domain face thermal infrared image respectively and the value of the target domain face visible light synthetic image Finally, the corresponding multi-layer local contrast loss is constructed.

[0070] The contrast loss function L is established based on the source domain human face thermal infrared image and the target domain human face visible light synthetic image. Con :

[0071]

[0072] Where τ = 0.07 is a hyperparameter, V is the anchor point of the target domain face visible light synthesis image, and H + and H - are a positive sample and N-1 negative samples in the source domain face thermal infrared image. The contrast loss is recorded as: L Con H .

[0073] Let one positive sample and N-1 negative samples also be selected from the target domain face visible light synthetic image, similar to the identity preservation loss, to ensure the similarity of the features and structures of G(H) and H, and prevent the generator from excessively changing the face synthetic image. Its loss function is denoted as: L Con G(H) .

[0074] Visible light synthesis image of target domain face I G(H) And the face visible light ground truth image I V (GT) extracts the corresponding face gradient map and Constructing the gradient boosting loss L Gm This gradient enhancement loss can reduce the generation of facial artifacts and maintain better facial contour information.

[0075]

[0076] Given that many thermal infrared-visible light datasets provide corresponding paired images in both domains, the unsupervised algorithm can be extended by enforcing additional constraints to minimize the "L1 distance" between the synthesized image and the real visible light image. This supervised loss is complementary to the unsupervised loss of CUT, and this additional regularization can supplement the unsupervised image synthesis algorithm. This loss is called pixel-level consistency loss and is denoted as: L Pcl . Among them I V is the corresponding visible light ground truth image (GT) of the face.

[0077] L pcl =||I G(H) -I V ||1

[0078] The adversarial loss between the generator G and the discriminator D, the face thermal infrared input image is represented as I H , the pseudo visible light image generated by the face generation network model is represented as I G(H) The skin color label information condition is represented by Z, which divides the skin color information into a one-hot encoding representation. During training, the category label and image data are cat-operated as input. (Face prior information condition) The face parsing graph is represented by P, and ζ is represented as the minimum L1 loss between the visible light synthetic image of the face synthesized by the generative network and the real face visible light image. Its generative network is based on the mapping relationship between face prior information and face label conditions: Therefore, its generated adversarial loss is:

[0079]

[0080] The total loss function of the model is:

[0081] L=λ1L Con H +λ2L Con G(H) +λ3L Pcl +λ4L Gm +λ5L gan

[0082] Among them, λ1, λ2, λ3, λ4, and λ5 are the hyperparameters of contrastive learning loss, identity-preserving contrastive learning loss, gradient enhancement loss, pixel-level consistency loss, and generative adversarial loss, respectively.

[0083] Within the framework of contrastive learning, this paper proposes a method that uses facial parsing graphs as prior knowledge to guide the generative network to learn local texture information of facial images. This method primarily introduces a spatial feature transformation mapping layer (STL), which uses the facial parsing graph mapping features as prior conditions to generate a pair of modulation parameters. Based on the modulation parameters, the facial features of the generative network are affine transformed, thereby adaptively optimizing the generation of facial images. Furthermore, the present invention designs a facial gradient enhancement loss to reduce the generation of facial artifacts. Furthermore, the present invention proposes adding skin color label conditional information to the input facial thermal infrared image and the paired visible light image, so that the image generation restores the corresponding skin color information as closely as possible.

[0084] The above embodiments further illustrate the purpose, technical solutions and advantages of the present invention in detail. It should be understood that the above embodiments are only preferred implementation plans of the present invention and are not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made to the present invention within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A method for converting thermal infrared to visible light images of faces based on prior information, characterized in that: include: Obtain an infrared face image to be converted and a corresponding visible light paired image, input the infrared face image into a trained face thermal infrared-visible light generation network model based on prior information, and obtain a visible light composite face image; the face thermal infrared-visible light generation network model based on prior information includes a face parsing graph conditional network module, a spatial feature transformation mapping layer, an attention module, a generator network module, and a discriminator; The process of training the face thermal infrared-visible light generation network model based on prior information includes: S1: Obtain a thermal infrared-visible light image dataset and classify the skin color information labels of the images in the dataset; S2: Preprocess the paired images in the dataset and input the preprocessed images into the face thermal infrared-visible light generation network model based on prior information; S3: Use the face parsing graph conditional network module to extract features from the preprocessed face parsing graph to obtain the face prior information features; S4: Using the generator network module to extract and encode the preprocessed facial thermal infrared image to obtain encoded facial feature information; S5: Inputting the facial prior information features and the encoded facial feature information into the spatial feature transformation mapping layer to generate a pair of modulation parameters; mapping the encoded facial feature information according to the modulation parameters to obtain mapped facial feature information; S6: The facial feature information undergoes multiple layers of residual transformation and decoder to obtain the corresponding visible light synthetic face image, which is then input into the discriminator for discrimination training. S7: Input the visible light composite face image and the thermal infrared face image into the encoder and two MLP mapping layers to obtain the corresponding thermal infrared face image features and visible light composite face image features; input the thermal infrared face image features and the visible light composite face image features into the attention module for contrastive learning, and calculate the model loss function according to the contrastive learning formula; S8: The Adam optimizer is used to optimize the model parameters. When the loss function of the model is minimized, the optimal parameters are output to obtain the optimal face thermal infrared-visible light generation network model based on prior information.

2. The method for converting thermal infrared to visible light images of faces based on prior information according to claim 1, characterized in that: The skin color information labels of the face images in the dataset are classified as follows: labels 0, 1, and 2 represent white, yellow, and black skin colors, respectively; the label information is added to the model for learning and one-hot encoding is used.

3. The method for converting thermal infrared to visible light images of faces based on prior information according to claim 1, characterized in that: The preprocessing of the paired images includes: face alignment of the image data in the dataset, locating the five key points of the face using the RetinaFace face detection algorithm, cropping the face image according to the located key points; and resizing the processed image to a 256×256 image.

4. The method for converting thermal infrared to visible light images of faces based on prior information according to claim 1, characterized in that: The face parsing image conditional network module includes three convolutional layers with 1×1 convolution kernels. The preprocessed face parsing image is subjected to three layers of convolution processing to obtain the face prior information features.

5. The method for converting thermal infrared to visible light images of faces based on prior information according to claim 1, characterized in that: The generator network module includes the encoder G enc , converter and decoder G dec ; Encoder G enc It is mainly composed of 3 layers of CIR. Each layer of CIR is composed of a convolution, InstanceNorm normalization, and ReLU activation function. The encoder extracts features from the input image. The converter is composed of 9 Residual blocks. Each Residual block is composed of a spatial feature transformation mapping layer STL and a CIR layer. This residual module mainly enhances the feature map extracted by the encoder. The decoder G dec It includes two CTIR layers, a Reflect operation layer, and a convolution layer. The CTIR layer is composed of deconvolution, InstanceNorm normalization, and ReLU activation function in sequence; the function of the decoder is upsampling operation, which gradually reconstructs the learned facial features to the original image size.

6. The method for converting thermal infrared to visible light images of faces based on prior information according to claim 1, characterized in that: The process of processing the input facial features using the spatial feature transformation mapping layer includes: performing two layers of convolution operations on the facial prior information features and the encoded facial feature information to obtain a pair of parameters α and β; combining the generated parameter pair with the feature output G of each layer of the generative network. F First do the dot multiplication, then do the addition operation to get the output of the entire STL network.

7. The method for converting thermal infrared to visible light images of faces based on prior information according to claim 1, characterized in that: The process of using the attention module to compare the thermal infrared input image of the face and the synthetic visible light image of the face includes: S71: Perform multi-layer feature extraction on the thermal infrared image features and the visible light composite image features of the face; that is, the thermal infrared image features and the visible light composite image features of the face are each passed through an encoder G enc and a two-layer MLP network layer H l Get the facial feature spectrum F H ∈R C×H×W and F V ∈R C×H×W ; S72: Reshape and transpose the characteristic spectrum of the face thermal infrared image to obtain a two-dimensional matrix Q H ∈R HW ×C and V H ∈R HW×C ; S73: According to the two-dimensional matrix Q H ∈R HW×C and V H ∈R HW×C Constructing a global attention contrastive loss.

8. The method for converting thermal infrared to visible light images of faces based on prior information according to claim 7, characterized in that: The process of constructing the global attention contrast loss includes: H Multiply by its transpose K H ∈R C×HW Get a matrix, perform Softmax normalization on each row of the matrix to get a global attention matrix A global ∈R HW×HW ; Calculate the entropy value H of each row in the global attention matrix according to the entropy calculation formula s And sort each row of data in the global attention matrix in ascending order according to the calculated entropy value; according to the sorted matrix, route the V in the source domain face thermal infrared image and the target domain face visible light synthetic image respectively. H ∈R HW×C and V V ∈R HW×C Features; finally route the corresponding value feature V H and V V To construct the global contrast loss.

9. The method for converting thermal infrared to visible light images of faces based on prior information according to claim 1, characterized in that: The loss function of the model is: L=λ1L Con H +λ2L Con G(H) +λ3L Pcl +λ4L Gm +λ5L gan Among them, λ1, λ2, λ3, λ4, and λ5 are the hyperparameters of contrastive learning loss, identity-preserving contrastive learning loss, gradient enhancement loss, pixel-level consistency loss, and generative adversarial loss, respectively.

Images