A Face Fusion Super-Resolution Method and System Based on Triple Unpaired Learning

Abstract

This application discloses a face fusion super-resolution method and system based on triplet unpaired learning. The method includes the following steps: preprocessing the face image in the original sketch photograph to obtain a training set and a test set; constructing a triplet face fusion super-resolution network model; inputting the training set into the constructed triplet face fusion super-resolution network model to obtain an optimized triplet face fusion super-resolution network model; inputting the test set into the optimized triplet face fusion super-resolution network model to reconstruct and enhance a high-resolution visible light face from paired low-resolution visible light face images and high-resolution sketch face images. This application utilizes the complementary information of visible light face and sketch face modalities to successfully solve the problems of fusing and super-resolution of two face modalities and the challenges of mismatched training data, generating high-quality face images for criminal case investigation.

Description

Technical Field

[0001] This application relates to the field of digital image technology, specifically to a face fusion super-resolution method and system based on triplet unpaired learning. Background Technology

[0002] In security and criminal investigation scenarios, visible light faces captured by surveillance cameras and sketches drawn by portrait experts are two important clues for identifying suspects. Visible light faces, acquired by the equipment, are objective, but are mostly low-resolution images due to the shooting environment; sketches contain detailed information, but are drawn manually and may introduce errors, resulting in significant differences from realistic facial modalities. Therefore, fusing and enhancing these two types of images based on their complementarity is a possible way to improve facial recognition accuracy. However, in practice, it is difficult to obtain paired training data for these three types of images: low-resolution visible light faces captured by cameras, sketches drawn by portrait experts, and high-resolution visible light faces of the target, making paired learning impossible.

[0003] With the development of deep learning in the field of computer vision, significant progress has been made in the research of image fusion algorithms, yielding remarkable results. In 2020, Xin et al. proposed a multi-frame face super-resolution method based on motion adaptive feedback units for fusing multi-frame face images; in 2022, Bai et al. proposed an identity-guided face generation algorithm that uses contour maps and high-quality identity maps to obtain fused faces; and in 2023, Cheng et al. proposed a general unsupervised image fusion network based on memory units.

[0004] Existing multi-frame face super-resolution methods are designed based on prior knowledge that the content of multiple frames is similar, which is not suitable for the fusion of visible light and sketch faces with large differences in content and modality. Reference-based face generation methods use high-quality face photos to guide face generation, but in security and criminal investigation scenarios, high-quality photos of relevant personnel cannot be used as reference images, which does not meet the actual needs. Existing general image fusion algorithms do not consider sketch and visible light image fusion scenarios. Summary of the Invention

[0005] The purpose of this application is to overcome the shortcomings of the above-mentioned background technology and provide a face fusion super-resolution method and system based on triplet unpaired learning.

[0006] Firstly, a face fusion super-resolution method based on triplet unpaired learning is provided, including the following steps:

[0007] The face images in the original sketch photos are preprocessed to obtain training and test sets. The training set consists of unpaired low-resolution visible light face images, high-resolution sketch face images, and high-resolution visible light face image datasets. The test set consists of paired low-resolution visible light face images and high-resolution sketch face image datasets.

[0008] Construct a triplet face fusion super-resolution network model;

[0009] Input the training set into the constructed triplet face fusion super-resolution network model to obtain the optimized triplet face fusion super-resolution network model;

[0010] The test set is input into the optimized triplet face fusion super-resolution network model, which reconstructs and enhances high-resolution visible light faces from paired low-resolution visible light face images and high-resolution sketch face images.

[0011] According to the first aspect, in the first implementation of the first aspect, in the step of constructing the triplet face fusion super-resolution network model, the triplet face fusion super-resolution network model includes a face fusion generator, a low-resolution visible light domain generator, a sketch domain generator, a high-resolution visible light domain discriminator, a low-resolution visible light domain discriminator, and a sketch domain discriminator.

[0012] According to the first implementation of the first aspect, in the second implementation of the first aspect, the face fusion generator includes a feature extraction unit, a feature fusion unit, and a feature reconstruction unit.

[0013] According to the first aspect, in the third implementation of the first aspect, the triplet face fusion super-resolution network model is a two-branch recurrent generative adversarial network, including a forward recurrent branch and a backward recurrent branch. The step of inputting the training set into the constructed triplet face fusion super-resolution network model and obtaining the optimized triplet face fusion super-resolution network model specifically includes the following steps:

[0014] The low-resolution visible light face image and high-resolution sketch face image from the training set are input into the forward loop branch for feature fusion processing to obtain a reconstructed and fused face.

[0015] The acquired first-reconstructed and fused face is processed by branch domain generation to obtain a second-reconstructed low-resolution visible light face and a second-reconstructed high-resolution sketch face.

[0016] The high-resolution visible light face images from the training set are input into the reverse loop branch for branch domain generation processing, resulting in a low-resolution visible light face reconstruction and a high-resolution sketch face reconstruction.

[0017] The acquired low-resolution visible light face and high-resolution sketch face from the first reconstruction are subjected to feature fusion processing to obtain a second-reconstructed fused face.

[0018] Based on the obtained primary reconstructed and fused face, secondary reconstructed low-resolution visible light face and secondary reconstructed high-resolution sketch face, primary reconstructed low-resolution visible light face and primary reconstructed high-resolution sketch face, secondary reconstructed and fused face, and training set, obtain the overall loss function of the model;

[0019] The training set is used to train the triplet face fusion super-resolution network model to obtain the trained triplet face fusion super-resolution network model.

[0020] According to the third implementation of the first aspect, in the fourth implementation of the first aspect, the step of inputting the low-resolution visible light face image and the high-resolution sketch face image of the training set into the forward loop branch, performing feature fusion processing, and obtaining a reconstructed fused face specifically includes the following steps:

[0021] The low-resolution visible light face images and high-resolution sketch face images from the training set are input into the feature extraction unit to obtain two basic features;

[0022] The two types of facial features are input into the feature fusion unit to obtain the fused features;

[0023] The acquired fusion features are input into the feature reconstruction unit to obtain a reconstructed fused face.

[0024] According to the third implementation of the first aspect, in the fifth implementation of the first aspect, the step of performing branch domain generation processing on the acquired primary reconstructed and fused face to obtain a secondary reconstructed low-resolution visible light face and a secondary reconstructed high-resolution sketch face specifically includes the following steps:

[0025] The acquired first-reconstruction face is input into a low-resolution visible light domain generator and a sketch domain generator to obtain a second-reconstruction low-resolution visible light face and a second-reconstruction high-resolution sketch face.

[0026] According to the third implementation of the first aspect, in the sixth implementation of the first aspect, the step of obtaining the discrimination loss and cycle consistency loss of each branch domain based on the acquired primary reconstructed fused face, secondary reconstructed low-resolution visible light face, secondary reconstructed high-resolution sketch face, primary reconstructed low-resolution visible light face, primary reconstructed high-resolution sketch face, secondary reconstructed fused face, and training set specifically includes the following steps:

[0027] Based on the acquired secondary reconstructed fused face, secondary reconstructed low-resolution visible light face, secondary reconstructed high-resolution sketch face, and training set, the cycle consistency loss of each branch domain is obtained.

[0028] Based on the acquired first-reconstruction fused face, first-reconstruction low-resolution visible light face, and first-reconstruction high-resolution sketch face, as well as the training set, the discrimination loss of each branch domain is obtained.

[0029] Based on the discriminative loss and cycle consistency loss of each branch domain, the overall loss function of the model is obtained.

[0030] Secondly, this application provides a face fusion super-resolution system based on triplet unpaired learning, including a face image preprocessing module, a model building module, a model optimization module, and a high-resolution visible light face acquisition module. The face image preprocessing module is used to preprocess the face images in the original sketch photographs to obtain a training set and a test set. The training set consists of unpaired low-resolution visible light face images, high-resolution sketch face images, and high-resolution visible light face image datasets, while the test set consists of paired low-resolution visible light face images and high-resolution sketch face image datasets. The model building module is used to construct a triplet face fusion super-resolution network model. The model optimization module is communicatively connected to the model building module and is used to input the training set into the constructed triplet face fusion super-resolution network model to obtain an optimized triplet face fusion super-resolution network model. The high-resolution visible light face acquisition module is communicatively connected to the face image preprocessing module and the model optimization module and is used to input the test set into the optimized triplet face fusion super-resolution network model to reconstruct and enhance a high-resolution visible light face from the paired low-resolution visible light face images and high-resolution sketch face images.

[0031] According to the second aspect, in the first implementation of the second aspect, the triplet face fusion super-resolution network model is a two-branch recurrent generative adversarial network, including a forward recurrent branch and a backward recurrent branch, and the model optimization module includes:

[0032] The first acquisition submodule inputs the low-resolution visible light face image and the high-resolution sketch face image from the training set into the forward loop branch, performs feature fusion processing, and obtains a reconstructed and fused face.

[0033] The second acquisition submodule performs branch domain generation processing on the acquired primary reconstructed and fused face to acquire a secondary reconstructed low-resolution visible light face and a secondary reconstructed high-resolution sketch face.

[0034] The third acquisition submodule inputs the high-resolution visible light face images of the training set into the reverse loop branch, performs branch domain generation processing, and obtains a reconstructed low-resolution visible light face and a reconstructed high-resolution sketch face.

[0035] The fourth acquisition submodule, which is communicatively connected to the third acquisition submodule, is used to perform feature fusion processing on the acquired first-reconstructed low-resolution visible light face and the first-reconstructed high-resolution sketch face to obtain a second-reconstructed fused face.

[0036] The model loss acquisition submodule is communicatively connected to the face image preprocessing module and the first acquisition submodule, the second acquisition submodule, the third acquisition submodule and the fourth acquisition submodule. It is used to acquire the overall model loss function based on the acquired primary reconstructed fused face, secondary reconstructed low-resolution visible light face and secondary reconstructed high-resolution sketch face, primary reconstructed low-resolution visible light face and primary reconstructed high-resolution sketch face, secondary reconstructed fused face and training set.

[0037] The model optimization submodule is communicatively connected to the face image preprocessing module and the model loss acquisition submodule. It is used to train the triplet face fusion super-resolution network model with the training set and obtain the trained triplet face fusion super-resolution network model.

[0038] According to the first implementation of the second aspect, in the second implementation of the second aspect, the first acquisition submodule includes:

[0039] The feature extraction unit is used to input the low-resolution visible light face image and the high-resolution sketch face image of the training set into the feature extraction unit to obtain two basic features;

[0040] The feature fusion unit is communicatively connected to the feature extraction unit and is used to input the two basic features obtained into the feature fusion unit to obtain fused features;

[0041] The feature reconstruction unit is communicatively connected to the feature fusion unit and is used to input the acquired fused features into the feature reconstruction unit to obtain a reconstructed fused face.

[0042] Compared with the prior art, the advantages of this application are as follows:

[0043] The face fusion super-resolution method based on triplet unpaired learning provided in this application utilizes complementary information from two modalities of faces: visible light faces and sketch faces. By constructing a triplet face fusion super-resolution network model for model training, optimization, and testing, it successfully solves the problems of fusing and super-resolution between two modalities of faces and the challenges of mispaired training data. This results in higher quality and more reliable high-resolution fused faces, enabling the use of low-quality images as reference images in security and criminal investigation scenarios to generate high-quality face images for criminal case investigation. Attached Figure Description

[0044] Figure 1 This is a flowchart of the face fusion super-resolution method based on triplet unpaired learning provided in the embodiments of this application;

[0045] Figure 2 This is a network diagram of the feature extraction unit in the face fusion super-resolution method based on triplet unpaired learning provided in the embodiments of this application;

[0046] Figure 3 This is a network diagram of the feature fusion unit in the face fusion super-resolution method based on triplet unpaired learning provided in the embodiments of this application;

[0047] Figure 4 This is a diagram of the face fusion generator network in the face fusion super-resolution method based on triplet unpaired learning provided in the embodiments of this application;

[0048] Figure 5 This is a flowchart of a super-resolution network framework for fusing low-resolution visible light and sketched faces based on triplet unpaired learning, provided in an embodiment of this application. Detailed Implementation

[0049] Specific embodiments of this application will now be described in detail, with examples of the application illustrated in the accompanying drawings. Although this application will be described in conjunction with specific embodiments, it will be understood that it is not intended to limit this application to the described embodiments. Rather, it is intended to cover variations, modifications, and equivalents included within the spirit and scope of this application as defined by the appended claims. It should be noted that the method steps described herein can be implemented by any functional block or functional arrangement, and any functional block or functional arrangement can be implemented as a physical entity or a logical entity, or a combination of both.

[0050] To enable those skilled in the art to better understand this application, the application will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0051] Note: The examples described below are merely specific examples and are not intended to limit the embodiments of this application to the specific steps, values, conditions, data, order, etc. Those skilled in the art can utilize the concepts of this application to construct more embodiments not mentioned herein by reading this specification.

[0052] In security and criminal investigation scenarios, high-quality facial photos cannot be obtained as reference images. It is necessary to obtain high-quality facial images from low-quality facial images. However, existing general image fusion algorithms do not consider sketch and visible light image fusion scenarios. Existing multi-frame face super-resolution methods are designed based on prior knowledge that the content of multiple frames is similar, which is not suitable for visible light and sketch face fusion with large differences in content and modality.

[0053] In view of this, see Figure 1 As shown, this application provides a face fusion super-resolution method based on triplet unpaired learning, including the following steps:

[0054] Step S1: Preprocess the face images in the original sketch photos to obtain training and test sets. The training set consists of unpaired low-resolution visible light face images, high-resolution sketch face images, and high-resolution visible light face image datasets. The test set consists of paired low-resolution visible light face images and high-resolution sketch face image datasets.

[0055] Step S2: Construct a triplet face fusion super-resolution network model;

[0056] Step S3: Input the training set into the constructed triplet face fusion super-resolution network model to obtain the optimized triplet face fusion super-resolution network model.

[0057] Step S4: Input the test set into the optimized triplet face fusion super-resolution network model to reconstruct and enhance the high-resolution visible light face by combining the paired low-resolution visible light face image and the high-resolution sketch face image.

[0058] The face fusion super-resolution method based on triplet unpaired learning provided in this application utilizes the complementary information of two face modalities, visible light face and sketch face, to fuse and enhance low-resolution visible light face and sketch face to high-resolution visible light domain. By constructing a triplet face fusion super-resolution network model for model training, optimization, and testing, it successfully solves the problems of fusing and super-resolution of two face modalities and training data mispairing, obtaining higher quality and more reliable high-resolution fused faces. This enables the use of low-quality images as reference images in security and criminal investigation scenarios to generate high-quality face images for criminal case investigation.

[0059] In one embodiment, in step S2, the step of constructing the triplet face fusion super-resolution network model, the triplet face fusion super-resolution network model includes a face fusion generator, a low-resolution visible light domain generator, a sketch domain generator, a high-resolution visible light domain discriminator, a low-resolution visible light domain discriminator, and a sketch domain discriminator. The low-resolution visible light domain generator and the sketch domain generator are composed of ResNet convolutional networks; the domain discriminator module consists of three convolutional network blocks with a kernel size of 4 and a stride of 2, and two convolutional network blocks with a kernel size of 4 and a stride of 1.

[0060] In one embodiment, the face fusion generator includes a feature extraction unit, a feature fusion unit, and a feature reconstruction unit, such as... Figure 2-4 As shown, the feature extraction unit is used to extract image features from low-resolution visible light face images and high-resolution sketch face images to obtain face image features of each modality; the feature fusion unit is used to fuse the visible light face image features and the sketch face image features to obtain fused features; the fused features are input into the feature reconstruction unit for feature reconstruction, and finally the reconstructed fused face is output.

[0061] This application provides a face fusion super-resolution method based on triplet unpaired learning. It utilizes complementary information from two face modalities—visible light face and sketch face—to fuse and enhance low-resolution visible light and sketch faces to a high-resolution visible light domain. A face fusion generator is embedded within a triplet face fusion super-resolution network framework. This network model is then trained, optimized, and tested. The method successfully solves the challenges of fusing and super-resolution two face modalities and addressing the issue of mismatched training data. This results in higher-quality, more reliable high-resolution fused faces, enabling the generation of high-quality face images from low-quality reference images for criminal investigations, particularly in security and criminal investigation scenarios.

[0062] In one embodiment, such as Figure 5 As shown, step S3, the triplet face fusion super-resolution network model, is a two-branch recurrent generative adversarial network, including a forward recurrent branch and a backward recurrent branch. The step of inputting the training set into the constructed triplet face fusion super-resolution network model to obtain the optimized triplet face fusion super-resolution network model specifically includes the following steps:

[0063] Step S31: Input the low-resolution visible light face image and high-resolution sketch face image from the training set into the forward loop branch, perform feature fusion processing, and obtain a reconstructed fused face.

[0064] Step S32: Perform branch domain generation processing on the acquired primary reconstructed and fused face to obtain a secondary reconstructed low-resolution visible light face and a secondary reconstructed high-resolution sketch face;

[0065] Step S33: Input the high-resolution visible light face images of the training set into the reverse loop branch, perform branch domain generation processing, and obtain a reconstructed low-resolution visible light face and a reconstructed high-resolution sketch face.

[0066] Step S34: Perform feature fusion processing on the acquired first-reconstructed low-resolution visible light face and the first-reconstructed high-resolution sketch face to obtain the second-reconstructed fused face;

[0067] Step S35: Based on the acquired first-reconstruction fused face, second-reconstruction low-resolution visible light face and second-reconstruction high-resolution sketch face, first-reconstruction low-resolution visible light face and first-reconstruction high-resolution sketch face, second-reconstruction fused face and training set, obtain the overall loss function of the model;

[0068] Step S36: Train the triplet face fusion super-resolution network model with the training set to obtain the trained triplet face fusion super-resolution network model.

[0069] In one embodiment, step S31, which involves inputting the low-resolution visible light face image and the high-resolution sketch face image from the training set into the forward loop branch for feature fusion processing to obtain a reconstructed and fused face, specifically includes the following steps:

[0070] Step S311: Input the low-resolution visible light face image and high-resolution sketch face image from the training set into the feature extraction unit to obtain two basic features;

[0071] Step S312: Input the two types of facial features obtained into the feature fusion unit to obtain the fused features;

[0072] Step S313: Input the acquired fusion features into the feature reconstruction unit to obtain a reconstructed fused face.

[0073] In a more specific embodiment, the feature extraction unit of the face fusion generator network includes a spatial attention module and a feature downsampling module. The spatial attention module consists of two 3×3 convolutional network blocks and an activation layer, and the feature downsampling module consists of a 7×7 convolutional network block and two 3×3 convolutional network blocks. The feature fusion unit consists of three fusion modules and six residual modules connected in series. Each fusion module includes a fusion branch, a visible light residual branch, and a sketch residual branch. Both residual branches consist of two 3×3 convolutional network blocks. The feature reconstruction unit is a residual module, which consists of two 3×3 convolutional network blocks.

[0074] The step of inputting low-resolution visible light face images and high-resolution sketch face images from the training set into the forward loop branch for feature fusion processing to obtain a reconstructed and fused face specifically includes the following steps:

[0075] Low-resolution visible light face images and high-resolution sketch face images are processed by the spatial attention module of the feature extraction unit to obtain spatial weights. The two images are multiplied by the spatial weights and then processed by the input feature downsampling module to extract features, resulting in two types of face features, including low-resolution visible light face features and high-resolution sketch face features.

[0076] The visible light residual branch enhances the low-resolution visible light face features, and the sketch residual branch enhances the sketch face features. The first fusion module uses a 1×1 convolutional network block in the fusion branch to fuse the low-resolution visible light and sketch face features obtained from the feature extraction unit. The i-th (i≥2) fusion module first uses a 1×1 convolutional network block in the fusion branch to perform the first fusion of the low-resolution visible light and sketch face features obtained from the previous fusion module, and then uses a 1×1 convolutional network block to fuse the fused features obtained from the previous fusion module with the first fusion result of this module.

[0077] The fusion features are fed into the residual module to obtain the final fusion enhancement features, which is the first reconstruction of the fused face.

[0078] In a more specific embodiment, step S31, inputting the low-resolution visible light face image and the high-resolution sketch face image from the training set into the forward loop branch, performing feature fusion processing, and obtaining a reconstructed and fused face step, specifically includes the following steps:

[0079] The low-resolution visible light face images from the training set I LR and high-resolution sketched human face images I Sk Each spatial attention module is input to obtain spatial attention weights. Each spatial attention module consists of two 3×3 Conv layers and a Sigmoid activation layer. Each Conv layer is followed by an InstanceNorm normalization layer. The process is as follows:

[0080] spa LR =Attention LR (I LR ), spa Sk =Attention Sk (I Sk )

[0081] Among them, Attention LR (I LR ) represents the input face image ILR Spatial attention module, Attention Sk (I Sk ) represents the input face image I Sk Spatial attention module, spa LR spa Sk The spatial attention weights obtained from the respective spatial attention modules are input into the low-resolution visible light face image and the high-resolution sketch face image, respectively.

[0082] The face image is multiplied by the corresponding spatial attention weights and input into the feature downsampling module to obtain two types of face features. The feature downsampling module consists of a 7×7 convolutional network block and two 3×3 convolutional network blocks. Each convolutional network block consists of a Conv (7×7 or 3×3), InstanceNorm, and ReLU concatenated. The process is as follows:

[0083]

[0084]

[0085] Among them, FExtra LR () and FExtra Sk () indicates the feature downsampling module. These are low-resolution visible light face features and high-resolution sketch face features obtained by the feature extraction unit.

[0086] Low-resolution visible light facial features and high-resolution sketch facial features are input into the feature fusion unit, and the two features are fused and enhanced to obtain the fused feature:

[0087] In this embodiment, each fusion module has a three-branch structure: a fusion branch, a visible light residual branch, and a sketch residual branch. These three branches are parallel. The visible light residual branch and the sketch residual branch are composed of 3×3Conv, InstanceNorm, ReLU, Drop, 3×3Conv, and InstanceNorm layers connected in series. The fusion network of the first fusion module consists of a Concat layer and a 1×1Conv layer. The first and second fusion networks of the i-th (i≥2) fusion module each consist of a Concat layer and a 1×1Conv layer.

[0088] Will and The first fusion module is input as follows:

[0089]

[0090]

[0091]

[0092] Where Fusion1() is the fusion computation, Res LR () represents the visible light residual calculation, Res Sk () represents the calculation of sketch residuals. The fusion features obtained from the first fusion module. and The first fusion module yields low-resolution visible light features and sketch features.

[0093] Will and Input the i-th (i≥2) fusion module. In this embodiment, i={2,3}, it is represented as follows:

[0094]

[0095]

[0096]

[0097] Fusion1() is the first fusion calculation, Fusion2() is the second fusion calculation, and Res LR () represents the visible light residual calculation, Res Sk () represents the calculation of sketch residuals. The fusion feature obtained by the i-th fusion module and These are the low-resolution visible light features and sketch features obtained by the i-th fusion module.

[0098] Will The input residual module undergoes further transformation and enhancement. Each residual module consists of a cascaded 3×3Conv, InstanceNorm, ReLU, Drop, 3×3Conv, and InstanceNorm layer, as shown in the following process:

[0099]

[0100] Where Res() is the residual calculation, F Fus To integrate and enhance facial features.

[0101] The feature reconstruction unit is used to fuse and enhance the acquired facial features F. Fus Reconstruction is performed to obtain a fused and enhanced face image I. HR The feature reconstruction unit consists of two 3×3 deconvolutional layers, one 7×7 Conv layer, and one Tanh layer.

[0102] In one embodiment, step S31, which involves inputting the low-resolution visible light face image and the high-resolution sketch face image from the training set into the forward loop branch for feature fusion processing to obtain a reconstructed and fused face, specifically includes the following steps:

[0103] Low-resolution visible light face images from the training set High-resolution sketched human face image The input forward loop branch first enters the face fusion generator, resulting in a reconstructed and fused face, which can be represented by the following formula:

[0104]

[0105] Among them, G Z This refers to a face fusion generator. This is a high-resolution domain fusion enhanced face generated from the input low-resolution visible light face and sketch face.

[0106] In one embodiment, step S32, which involves performing branch domain generation processing on the acquired primary reconstructed and fused face to obtain a secondary reconstructed low-resolution visible light face and a secondary reconstructed high-resolution sketch face, specifically includes the following steps:

[0107] The obtained first-reconstructed face is input into a low-resolution visible light domain generator and a sketch generator to obtain a second-reconstructed low-resolution visible light domain face and a second-reconstructed high-resolution sketch face, as shown in the following formula:

[0108]

[0109] Among them, G X For low-resolution visible light domain generators, G Y For sketch domain generator, To use a low-resolution visible light domain generator to reconstruct and fuse faces in one go Low-resolution visible light face after conversion to low-resolution domain To use the sketch domain generator to reconstruct and blend a face in one go The sketched face after being converted to the sketch field.

[0110] In one embodiment, step S33, inputting the high-resolution visible light face images from the training set into the reverse loop branch, performing branch domain generation processing to obtain a reconstructed low-resolution visible light face and a reconstructed high-resolution sketch face, specifically includes the following steps:

[0111] The high-resolution visible light face image dataset from the training set is input into the reverse loop branch. First, it enters the low-resolution visible light domain generator and the sketch domain generator respectively to obtain a reconstructed low-resolution visible light face and a reconstructed high-resolution sketch face, as shown in the following formula:

[0112]

[0113] Among them, G X For low-resolution visible light domain generators, G Y For sketch domain generator, To use a low-resolution visible light domain generator to generate high-resolution visible light faces Low-resolution visible light face after conversion to low-resolution domain To use the sketch domain generator Sketch face after conversion to sketch field

[0114] In one embodiment, step S34, which involves fusing the acquired low-resolution visible light face and high-resolution sketch face from the first reconstruction to obtain the second-reconstructed fused face, specifically includes the following steps:

[0115] The low-resolution visible light face reconstructed in the first stage and the high-resolution sketch face reconstructed in the first stage are input into the face fusion generator to obtain the fused face reconstructed in the second stage, as shown in the following formula:

[0116]

[0117] Among them G Z This refers to a face fusion generator. This is a fused face generated by reconstructing a low-resolution visible light face once and a high-resolution sketch face once.

[0118] In one embodiment, step S35, which involves obtaining the discrimination loss and cycle consistency loss for each branch domain based on the acquired primary reconstructed fused face, secondary reconstructed low-resolution visible light face, secondary reconstructed high-resolution sketch face, primary reconstructed low-resolution visible light face, primary reconstructed high-resolution sketch face, secondary reconstructed fused face, and training set, specifically includes the following steps:

[0119] Step S351: Input the acquired secondary reconstructed fused face, secondary reconstructed low-resolution visible light face, and secondary reconstructed high-resolution sketch face, along with the training set, into the following formula to calculate the cycle consistency loss for each branch domain:

[0120]

[0121]

[0122]

[0123] in, The original input is a low-resolution visible light face. This is a low-resolution visible light face after secondary reconstruction and regression. For two types of low-resolution visible light face images, pixel loss is considered. The original input is a sketched human face. VGG is used to create a sketched face after regression in the secondary reconstruction domain. l () indicates that the l-th layer of the VGG network is selected as the feature output layer for loss calculation, and the pre-trained VGG network is used to extract the features respectively. and Features The perceptual loss of features in two types of sketched human faces; The original input is a high-resolution visible light face. To obtain the high-resolution fused face after secondary reconstruction domain regression, a pre-trained VGG network was used to extract the following: and Features This is the sum of pixel loss and perceptual loss in features for two high-resolution visible light faces.

[0124] Step S352: Based on the acquired reconstructed and fused face, reconstructed low-resolution visible light face, and reconstructed high-resolution sketch face, as well as the training set, obtain the discrimination loss for each branch domain. This specifically includes the following steps:

[0125] The high-resolution visible light face and the first-reconstruction fused face from the training set are fed into the high-resolution visible light domain discriminator. The discrimination loss is calculated separately and the network parameters are updated, as shown in the following figure:

[0126]

[0127] Among them, G Z D represents a face fusion generator. Z This represents a high-resolution visible light domain discriminator, where x, y, and z represent low-resolution visible light face models, respectively. Sketch of a human face High-resolution visible light face logD Z (z) represents a true high-resolution visible light image. The log probability obtained from the input discriminator, log(1-D) z (G Z (x,y) represents the reconstructed and fused image obtained after the generator. The log probability obtained from the input discriminator This indicates that training the high-resolution visible light domain discriminator requires maximizing... Training the face fusion generator requires minimizing

[0128] The low-resolution visible light faces from the training set and the reconstructed low-resolution visible light faces from the first reconstruction are fed into the low-resolution visible light domain discriminator. The discrimination loss is calculated and the network parameters are updated, as shown below:

[0129]

[0130] Among them, G X D represents a low-resolution visible light domain generator. X This represents a low-resolution visible light domain discriminator, where x and z represent low-resolution visible light face models, respectively. High-resolution visible light face logD t (x) represents a true low-resolution visible light image. The log probability obtained from the input discriminator, log(1-D) X (G X (z))) represents a pseudo-low-resolution visible light image reconstructed in one step. The log probability obtained from the input discriminator This indicates that training the low-resolution visible light domain discriminator requires maximizing... Training a low-resolution visible light domain generator requires minimizing

[0131] The high-resolution sketch faces from the training set and the first-order reconstructed high-resolution sketch faces are fed into the sketch domain discriminator. The discrimination loss is calculated and the network parameters are updated accordingly, as shown below:

[0132]

[0133] Among them, G y d represents the sketch domain generator. y This represents the sketch domain discriminator, where y and z represent the sketched human face, respectively. High-resolution visible light face logd t (y) represents the actual sketch image. The log probability obtained from the input discriminator, log(1-D) Y (G Y (z))) represents a fake image reconstructed in one step. The log probability obtained from the input discriminator This indicates that the training of the sketch domain discriminator during training needs to maximize... Training the sketch domain generator needs to minimize

[0134] Step S353: Based on the discriminative loss and cycle consistency loss of each branch domain, obtain the overall model loss function, as shown in the following formula:

[0135]

[0136] Where λ represents the empirical weight settings between the various losses, λ1 is the empirical weight in the low-resolution visible light domain, λ2 is the empirical weight in the low-resolution sketch domain, and λ3 is the empirical weight in the low-resolution sketch domain.

[0137] For high-resolution visible light domain empirical weights, These represent the cycle consistency loss in the low-resolution visible light domain, the sketch domain, and the high-resolution visible light domain, respectively. These represent the discrimination losses in the low-resolution visible light domain, the sketch domain, and the high-resolution visible light domain, respectively. In this embodiment, the preferred values ​​for each weight are λ1 = 10, λ2 = 0.01, and λ3 = [10, 0.01]. It consists of two parts: pixel loss and perceptual loss. Its loss weight λ3 has two values, which can be adjusted as needed during specific implementation.

[0138] In one embodiment, step S36, training the triplet face fusion super-resolution network model with the training set to obtain the trained triplet face fusion super-resolution network model, specifically includes the following steps:

[0139] By adjusting the empirical weights of each branch domain in the model's loss function until L total Converge and obtain the trained triplet face fusion super-resolution network model.

[0140] In this application, each branch domain includes a low-resolution visible light domain, a low-resolution sketch domain, and a high-resolution visible light domain.

[0141] Based on the same inventive concept, this application provides a face fusion super-resolution system based on triplet unpaired learning, including a face image preprocessing module, a model building module, a model optimization module, and a high-resolution visible light face acquisition module.

[0142] The face image preprocessing module preprocesses the face images in the original sketch photographs to obtain training and test sets. The training set consists of unpaired low-resolution visible light face images, high-resolution sketch face images, and high-resolution visible light face image datasets. The test set consists of paired low-resolution visible light face images and high-resolution sketch face image datasets. The model building module constructs a triplet face fusion super-resolution network model. The model optimization module is communicatively connected to the model building module and is used to input the training set into the constructed triplet face fusion super-resolution network model to obtain the optimized triplet face fusion super-resolution network model. The high-resolution visible light face acquisition module is communicatively connected to the face image preprocessing module and the model optimization module. It is used to input the test set into the optimized triplet face fusion super-resolution network model to reconstruct and enhance a high-resolution visible light face from the paired low-resolution visible light face images and high-resolution sketch face images.

[0143] In one embodiment, the triplet face fusion super-resolution network model is a two-branch recurrent generative adversarial network, including a forward recurrent branch and a backward recurrent branch. The model optimization module includes:

[0144] The first acquisition submodule inputs the low-resolution visible light face image and the high-resolution sketch face image from the training set into the forward loop branch, performs feature fusion processing, and obtains a reconstructed and fused face.

[0145] The second acquisition submodule performs branch domain generation processing on the acquired primary reconstructed and fused face to acquire a secondary reconstructed low-resolution visible light face and a secondary reconstructed high-resolution sketch face.

[0146] The third acquisition submodule inputs the high-resolution visible light face images of the training set into the reverse loop branch, performs branch domain generation processing, and obtains a reconstructed low-resolution visible light face and a reconstructed high-resolution sketch face.

[0147] The fourth acquisition submodule, which is communicatively connected to the third acquisition submodule, is used to perform feature fusion processing on the acquired first-reconstructed low-resolution visible light face and the first-reconstructed high-resolution sketch face to obtain a second-reconstructed fused face.

[0148] The model loss acquisition submodule is communicatively connected to the face image preprocessing module and the first acquisition submodule, the second acquisition submodule, the third acquisition submodule and the fourth acquisition submodule. It is used to acquire the overall model loss function based on the acquired primary reconstructed fused face, secondary reconstructed low-resolution visible light face and secondary reconstructed high-resolution sketch face, primary reconstructed low-resolution visible light face and primary reconstructed high-resolution sketch face, secondary reconstructed fused face and training set.

[0149] The model optimization submodule is communicatively connected to the face image preprocessing module and the model loss acquisition submodule. It is used to train the triplet face fusion super-resolution network model with the training set and obtain the trained triplet face fusion super-resolution network model.

[0150] In one embodiment, the first acquisition submodule specifically includes the following steps:

[0151] The feature extraction unit is used to input the low-resolution visible light face image and the high-resolution sketch face image of the training set into the feature extraction unit to obtain two basic features;

[0152] The feature fusion unit is communicatively connected to the feature extraction unit and is used to input the two basic features obtained into the feature fusion unit to obtain fused features;

[0153] The feature reconstruction unit is communicatively connected to the feature fusion unit and is used to input the acquired fused features into the feature reconstruction unit to obtain a reconstructed fused face.

[0154] Based on the same inventive concept, embodiments of this application also provide a computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements all or part of the method steps of the above method.

[0155] This application implements all or part of the processes in the above methods, which can also be accomplished by a computer program instructing related hardware. The computer program can be stored in a computer-readable storage medium, and when executed by a processor, it can implement the steps provided in the various method embodiments described above. The computer program includes computer program code, which can be in the form of source code, object code, executable files, or certain intermediate forms. The computer-readable medium can include: any entity or device capable of carrying computer program code, recording media, USB flash drives, portable hard drives, magnetic disks, optical disks, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signals, telecommunication signals, and software distribution media, etc. It should be noted that the content included in the computer-readable medium can be appropriately added or removed according to the requirements of legislation and patent practice in the jurisdiction. For example, in some jurisdictions, according to legislation and patent practice, the computer-readable medium does not include electrical carrier signals and telecommunication signals.

[0156] Based on the same inventive concept, embodiments of this application also provide an electronic device, including a memory and a processor. The memory stores a computer program that runs on the processor. When the processor executes the computer program, it implements all or part of the method steps described above.

[0157] The processor can be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor can be a microprocessor or any conventional processor. The processor is the control center of the computer device, connecting all parts of the computer device through various interfaces and lines.

[0158] Memory can be used to store computer programs and / or modules. The processor performs various functions of the computer device by running or executing the computer programs and / or modules stored in the memory, and by accessing data stored in the memory. Memory can primarily include a program storage area and a data storage area. The program storage area can store the operating system and at least one application program required for a function (e.g., sound playback, image playback, etc.); the data storage area can store data created based on the use of the mobile phone (e.g., audio data, video data, etc.). Furthermore, memory can include high-speed random access memory, and can also include non-volatile memory, such as hard disks, RAM, plug-in hard disks, SmartMedia Cards (SMC), Secure Digital (SD) cards, Flash Cards, at least one disk storage device, flash memory device, or other volatile solid-state storage devices.

[0159] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, servers, or computer program products. Therefore, this application can be provided in the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can be in the form of a computer program product implemented on one or more computer-usable storage media (including, but not limited to, disk storage and optical storage) containing computer-usable program code.

[0160] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), servers, and computer program products provided according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0161] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0162] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0163] Obviously, those skilled in the art can make various modifications and variations to this application without departing from the spirit and scope of this application. Therefore, if such modifications and variations fall within the scope of the claims of this application and their equivalents, this application also intends to include such modifications and variations.

Claims

1. A face fusion super-resolution method based on triplet unpaired learning, characterized in that, Includes the following steps: The face images in the original sketch photos are preprocessed to obtain training and test sets. The training set consists of unpaired low-resolution visible light face images, high-resolution sketch face images, and high-resolution visible light face image datasets. The test set consists of paired low-resolution visible light face images and high-resolution sketch face image datasets. Construct a triplet face fusion super-resolution network model; Input the training set into the constructed triplet face fusion super-resolution network model to obtain the optimized triplet face fusion super-resolution network model; The test set is input into the optimized triplet face fusion super-resolution network model, which reconstructs and enhances high-resolution visible light faces from paired low-resolution visible light face images and high-resolution sketch face images. The triplet face fusion super-resolution network model is a two-branch recurrent generative adversarial network, including a forward recurrent branch and a backward recurrent branch. The step of inputting the training set into the constructed triplet face fusion super-resolution network model and obtaining the optimized triplet face fusion super-resolution network model specifically includes the following steps: The low-resolution visible light face image and high-resolution sketch face image from the training set are input into the forward loop branch for feature fusion processing to obtain a reconstructed and fused face. The acquired first-reconstructed and fused face is processed by branch domain generation to obtain a second-reconstructed low-resolution visible light face and a second-reconstructed high-resolution sketch face. The high-resolution visible light face images from the training set are input into the reverse loop branch for branch domain generation processing, resulting in a low-resolution visible light face reconstruction and a high-resolution sketch face reconstruction. The acquired low-resolution visible light face and high-resolution sketch face from the first reconstruction are subjected to feature fusion processing to obtain a second-reconstructed fused face. Based on the obtained primary reconstructed and fused face, secondary reconstructed low-resolution visible light face and secondary reconstructed high-resolution sketch face, primary reconstructed low-resolution visible light face and primary reconstructed high-resolution sketch face, secondary reconstructed and fused face, and training set, obtain the overall loss function of the model; The training set is used to train the triplet face fusion super-resolution network model to obtain the trained triplet face fusion super-resolution network model.

2. The face fusion super-resolution method based on triplet unpaired learning as described in claim 1, characterized in that, In the step of constructing the triplet face fusion super-resolution network model, the triplet face fusion super-resolution network model includes a face fusion generator, a low-resolution visible light domain generator, a sketch domain generator, a high-resolution visible light domain discriminator, a low-resolution visible light domain discriminator, and a sketch domain discriminator.

3. The face fusion super-resolution method based on triplet unpaired learning as described in claim 2, characterized in that, The face fusion generator includes a feature extraction unit, a feature fusion unit, and a feature reconstruction unit.

4. The face fusion super-resolution method based on triplet unpaired learning as described in claim 1, characterized in that, The step of inputting low-resolution visible light face images and high-resolution sketch face images from the training set into the forward loop branch for feature fusion processing to obtain a reconstructed and fused face specifically includes the following steps: The low-resolution visible light face images and high-resolution sketch face images from the training set are input into the feature extraction unit to obtain two basic features; The two types of facial features are input into the feature fusion unit to obtain the fused features; The acquired fusion features are input into the feature reconstruction unit to obtain a reconstructed fused face.

5. The face fusion super-resolution method based on triplet unpaired learning as described in claim 1, characterized in that, The step of performing branch domain generation processing on the acquired primary reconstructed and fused face to obtain a secondary reconstructed low-resolution visible light face and a secondary reconstructed high-resolution sketch face specifically includes the following steps: The acquired first-reconstruction face is input into a low-resolution visible light domain generator and a sketch generator to obtain a second-reconstruction low-resolution visible light face and a second-reconstruction high-resolution sketch face.

6. The face fusion super-resolution method based on triplet unpaired learning as described in claim 1, characterized in that, The step of obtaining the discrimination loss and cycle consistency loss for each branch domain based on the acquired primary reconstructed and fused face, secondary reconstructed low-resolution visible light face, secondary reconstructed high-resolution sketch face, primary reconstructed low-resolution visible light face, primary reconstructed high-resolution sketch face, secondary reconstructed and fused face, and training set specifically includes the following steps: Based on the acquired first-reconstruction fused face, first-reconstruction low-resolution visible light face, and first-reconstruction high-resolution sketch face, as well as the training set, the discrimination loss of each branch domain is obtained. Based on the acquired secondary reconstructed fused face, secondary reconstructed low-resolution visible light face, secondary reconstructed high-resolution sketch face, and training set, the cycle consistency loss of each branch domain is obtained. Based on the discriminative loss and cycle consistency loss of each branch domain, the overall loss function of the model is obtained.

7. A face fusion super-resolution system based on triple unpaired learning, characterized in that, include: The face image preprocessing module is used to preprocess the face images in the original sketch photos to obtain training and test sets. The training set consists of unpaired low-resolution visible light face images, high-resolution sketch face images, and high-resolution visible light face image datasets, while the test set consists of paired low-resolution visible light face images and high-resolution sketch face image datasets. The model building module is used to build a triplet face fusion super-resolution network model. The model optimization module is communicatively connected to the model construction module and is used to input the training set into the constructed triplet face fusion super-resolution network model to obtain the optimized triplet face fusion super-resolution network model. The high-resolution visible light face acquisition module is communicatively connected to the face image preprocessing module and the model optimization module. It is used to input the test set into the optimized triplet face fusion super-resolution network model and reconstruct and enhance the high-resolution visible light face by combining the paired low-resolution visible light face image and the high-resolution sketch face image. The triplet face fusion super-resolution network model is a two-branch recurrent generative adversarial network, including a forward recurrent branch and a backward recurrent branch. The model optimization module includes: The first acquisition submodule inputs the low-resolution visible light face image and the high-resolution sketch face image from the training set into the forward loop branch, performs feature fusion processing, and obtains a reconstructed and fused face. The second acquisition submodule performs branch domain generation processing on the acquired primary reconstructed and fused face to acquire a secondary reconstructed low-resolution visible light face and a secondary reconstructed high-resolution sketch face. The third acquisition submodule inputs the high-resolution visible light face images of the training set into the reverse loop branch, performs branch domain generation processing, and obtains a reconstructed low-resolution visible light face and a reconstructed high-resolution sketch face. The fourth acquisition submodule, which is communicatively connected to the third acquisition submodule, is used to perform feature fusion processing on the acquired first-reconstructed low-resolution visible light face and the first-reconstructed high-resolution sketch face to obtain a second-reconstructed fused face. The model loss acquisition submodule is communicatively connected to the face image preprocessing module and the first acquisition submodule, the second acquisition submodule, the third acquisition submodule and the fourth acquisition submodule. It is used to acquire the overall model loss function based on the acquired primary reconstructed fused face, secondary reconstructed low-resolution visible light face and secondary reconstructed high-resolution sketch face, primary reconstructed low-resolution visible light face and primary reconstructed high-resolution sketch face, secondary reconstructed fused face and training set. The model optimization submodule is communicatively connected to the face image preprocessing module and the model loss acquisition submodule. It is used to train the triplet face fusion super-resolution network model with the training set and obtain the trained triplet face fusion super-resolution network model.

8. The face fusion super-resolution system based on triplet unpaired learning as described in claim 7, characterized in that, The first acquisition submodule includes: The feature extraction unit is used to input the low-resolution visible light face image and the high-resolution sketch face image of the training set into the feature extraction unit to obtain two basic features; The feature fusion unit is communicatively connected to the feature extraction unit and is used to input the two basic features obtained into the feature fusion unit to obtain fused features; The feature reconstruction unit is communicatively connected to the feature fusion unit and is used to input the acquired fused features into the feature reconstruction unit to obtain a reconstructed fused face.

Images