A face image reconstruction method and device, and a storage medium

Abstract

The application provides a face image reconstruction method and device and a storage medium, and belongs to the field of image reconstruction. The method comprises the following steps: obtaining a super-resolution training model by sequentially training and analyzing super-resolution images of a super-resolution to-be-trained model through each original face image; obtaining target super-resolution images by respectively identifying the images of the original face images through the super-resolution training model; obtaining a face reconstruction training model by training and analyzing face reconstruction images of a face reconstruction to-be-trained model through all original face images and all target super-resolution images; and obtaining a face image reconstruction result by respectively reconstructing the images of the target super-resolution images through the face reconstruction training model. The application can reconstruct HR face images with high fidelity and identity perception, can extract more detailed information, and improves visual fidelity.

Description

Technical Field

[0001] This invention relates to the field of image reconstruction technology, specifically to a method, apparatus, and storage medium for reconstructing human face images. Background Technology

[0002] In recent years, contrastive learning methods based on instance discrimination as a pre-task have been widely used in unsupervised representation learning. This method learns visual representations by making views from the same instance similar and views from different instances dissimilar. The learned visual representations can be applied to various downstream tasks, especially high-level tasks such as image clustering, knowledge distillation, and supervised image classification, and can achieve the desired results.

[0003] The principle of contrastive learning is to increase the mutual information in the learned representation by bringing positive samples closer to the anchor point and pushing negative samples away in the representation space. The choice of positive and negative samples depends on the specific downstream task; for example, augmentations of the original data may be considered positive samples, or multiple views of the same sample may be considered positive samples. Directly applying contrastive learning methods to low-level image processing tasks presents several challenges. First, the learned global visual representation is not suitable for low-level tasks that require rich texture and contextual information. Second, although a series of data augmentations have been proposed to generate positive and negative samples for high-level downstream tasks, most complex data augmentations, apart from some simple geometric augmentations, fail to maintain dense pixel correspondences and are therefore unsuitable for low-level tasks. Third, contrastive loss requires a meaningful embedding space. Compared to high-level tasks that attempt to obtain optimal semantic representations, low-level tasks aim to reconstruct the recovered result in the data space. Exploring a suitable and meaningful embedding space in which the contrastive loss can be effectively defined is crucial.

[0004] Current contrastive learning-based methods for low-level tasks mainly focus on utilizing negative samples while treating real images as positive samples. For example, Wu et al. treated degraded images as negative samples and proposed a novel image dehazing method with contrast regularization. Wang et al. used other examples in the dataset as negative samples for image super-resolution and underwater image restoration. These methods demonstrate the effectiveness of incorporating contrast constraints into low-level tasks. Another research direction is to model the statistical features of images through contrastive learning. Dong et al. assumed that two image patches from the same sample have similar noise distributions, while two image patches from two different samples have two different noise distributions, and proposed a residual contrastive loss for joint demosaicing and denoising. Wang et al. applied contrastive loss to pre-train a kernel estimation model that aims to separate different degradations and obtain different degradation-aware representations. Zhang et al. proposed a contrastive learning strategy in the feature channel space to obtain resolution-invariant features. They used feature maps from different channels as samples and assumed that the corresponding channels of the LR and HR feature maps are positive, while feature maps from different channels are negative. One research direction among these methods defines positive samples as the original image, while negative samples are simply defined as suboptimal or other images in the dataset. Although these negative samples differ from the reconstructed image, they are easily distinguishable, i.e., they are too far apart to contribute to the contrastive loss. Another research direction attempts to generate some global image features unaffected by noise based on contrastive learning, but these methods neglect to construct effective positive and negative samples for the reconstructed image. Furthermore, since the contrastive loss of these methods is defined in some specific embedding space, it cannot be well generalized to other methods.

[0005] Current deep learning-based face super-resolution methods often produce smooth but uncertain results that appear unnatural and unreliable, ignoring the unique texture structure of the face. Because the mapping from LR to HR is not uniquely determined and may exist in multiple function spaces, the super-resolution result may be the average of all possible outputs from the SR network. Summary of the Invention

[0006] The technical problem to be solved by the present invention is to address the shortcomings of the prior art by providing a method, apparatus and storage medium for facial image reconstruction.

[0007] The technical solution of the present invention to solve the above-mentioned technical problems is as follows: A face image reconstruction method, comprising the following steps:

[0008] Import multiple original face images and construct a super-resolution training model. Then, use each of the original face images to sequentially perform super-resolution image training analysis on the super-resolution training model to obtain a super-resolution training model corresponding to each of the original face images.

[0009] By performing image recognition on each of the original face images using a super-resolution training model corresponding to each of the original face images, a target super-resolution image corresponding to each of the original face images is obtained.

[0010] A face reconstruction training model is constructed by performing face reconstruction image training analysis on the face reconstruction training model using all original face images and all target super-resolution images to obtain a face reconstruction training model.

[0011] The face reconstruction training model is used to reconstruct each of the target super-resolution images to obtain face reconstruction images corresponding to each of the original face images, and all face reconstruction images are used as the face image reconstruction results.

[0012] Another technical solution of the present invention to solve the above-mentioned technical problems is as follows: A face image reconstruction device, comprising:

[0013] The first training and analysis module is used to import multiple original face images and construct a super-resolution model to be trained. The super-resolution model to be trained is then trained and analyzed using each of the original face images to obtain a super-resolution training model corresponding to each of the original face images.

[0014] The image recognition module is used to perform image recognition on each of the original face images by using a super-resolution training model corresponding to each of the original face images, so as to obtain a target super-resolution image corresponding to each of the original face images;

[0015] The second training and analysis module is used to construct a face reconstruction training model. It performs face reconstruction image training analysis on the face reconstruction training model using all original face images and all target super-resolution images to obtain a face reconstruction training model.

[0016] The face image reconstruction result module is used to reconstruct each of the target super-resolution images using the face reconstruction training model, to obtain face reconstruction images corresponding to each of the original face images, and to use all the face reconstruction images as the face image reconstruction result.

[0017] Another technical solution of the present invention to solve the above-mentioned technical problems is as follows: a face image reconstruction device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements the face image reconstruction method as described above.

[0018] Another technical solution of the present invention to solve the above-mentioned technical problems is as follows: a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the face image reconstruction method as described above.

[0019] The beneficial effects of this invention are as follows: a super-resolution training model is obtained by sequentially training and analyzing the super-resolution images of the super-resolution training model to be trained using the original face images; a target super-resolution image is obtained by image recognition of the original face images using the super-resolution training model; a face reconstruction training model is obtained by training and analyzing the face reconstruction images of the face reconstruction training model to be trained using the original face images and the target super-resolution images; and a face image reconstruction result is obtained by reconstructing the target super-resolution image using the face reconstruction training model. This invention can reconstruct high-fidelity and identity-aware HR face images, extract more detailed information, and improve visual fidelity. Attached Figure Description

[0020] Figure 1 A schematic flowchart of a face image reconstruction method provided in an embodiment of the present invention;

[0021] Figure 2 This is a block diagram of a face image reconstruction device provided in an embodiment of the present invention. Detailed Implementation

[0022] The principles and features of the present invention are described below with reference to the accompanying drawings. The examples given are only for explaining the present invention and are not intended to limit the scope of the present invention.

[0023] Figure 1 This is a flowchart illustrating a face image reconstruction method provided in an embodiment of the present invention.

[0024] like Figure 1 As shown, a face image reconstruction method includes the following steps:

[0025] Import multiple original face images and construct a super-resolution training model. Then, use each of the original face images to sequentially perform super-resolution image training analysis on the super-resolution training model to obtain a super-resolution training model corresponding to each of the original face images.

[0026] By performing image recognition on each of the original face images using a super-resolution training model corresponding to each of the original face images, a target super-resolution image corresponding to each of the original face images is obtained.

[0027] A face reconstruction training model is constructed by performing face reconstruction image training analysis on the face reconstruction training model using all original face images and all target super-resolution images to obtain a face reconstruction training model.

[0028] The face reconstruction training model is used to reconstruct each of the target super-resolution images to obtain face reconstruction images corresponding to each of the original face images, and all face reconstruction images are used as the face image reconstruction results.

[0029] It should be understood that existing high-quality face datasets FFHQ images (i.e., multiple original face images) are used, which include facial attributes such as age and skin color.

[0030] It should be understood that, based on contrastive learning, the identity feature representation of the face recognition network (i.e., the face reconstruction training model) is used as a supervision signal, and the identity features are used to further constrain the results of contrastive learning in order to obtain higher quality face reconstruction images.

[0031] Specifically, this invention uses the FFHQ dataset (i.e., multiple original face images), a high-quality face dataset containing facial attributes such as age and skin color. 4000 images are randomly selected, with 3800 images used as the training dataset, 100 images as the validation dataset, and 100 images as the test dataset. The scaling factor is ×4, and all HR images are resized to 256×256. Evaluation metrics used include PSNR, SSIM, and LPIPS. Experiments are conducted in a PyTorch environment using two Nvidia RTX 2080 Ti GPUs. Furthermore, the model is trained using the ADAM optimizer with parameters β1 = 0.9, β2 = 0.999, and e = 10. -8 The training iterations were performed 2000 times, and the batch size used was 16.

[0032] It should be understood that datasets are prepared as input to the algorithm and to verify the algorithm's performance.

[0033] In the above embodiments, a super-resolution training model is obtained by sequentially training and analyzing the super-resolution images of the super-resolution training model using the original face images. The target super-resolution image is obtained by image recognition of the original face images using the super-resolution training model. The face reconstruction training model is obtained by training and analyzing the face reconstruction images of the face reconstruction training model using the original face images and the target super-resolution images. The face reconstruction training model is then used to reconstruct the face image of the target super-resolution image to obtain the face image reconstruction result. This method can reconstruct high-fidelity and identity-aware HR face images, extract more detailed information, and improve visual fidelity.

[0034] Optionally, as an embodiment of the present invention, the super-resolution training model includes a pair of sequentially arranged upsampling blocks and a pair of sequentially arranged downsampling blocks. The process of sequentially performing super-resolution image training analysis on the super-resolution training model using each of the original face images to obtain a super-resolution training model corresponding to each of the original face images includes:

[0035] S11: The upsampling block sequentially performs upsampling analysis on each of the original face images to obtain upsampled face images corresponding to each of the original face images;

[0036] S12: The downsampling block is used to perform downsampling analysis on each of the original face images in sequence to obtain the original face super-resolution image corresponding to each of the original face images;

[0037] S13: Image enhancement processing is performed on each of the original face super-resolution images sequentially through multiple preset filters to obtain a positive sample set corresponding to each of the original face images;

[0038] S14: The original super-resolution face images are degraded sequentially using multiple preset filters to obtain a negative sample set corresponding to each original face image;

[0039] S15: Analyze the contrastive learning loss values ​​of each of the original face super-resolution images, the positive sample set corresponding to the original face image, and the negative sample set corresponding to the original face image in sequence to obtain the contrastive learning loss value corresponding to each of the original face images.

[0040] S16: Sequentially determine whether each of the contrastive learning loss values ​​is greater than a preset first loss threshold. If yes, update the parameters of the super-resolution training model according to the contrastive learning loss value, and use the updated super-resolution training model as the super-resolution training model corresponding to the original face image, and return to step S11; if no, use the super-resolution training model as the super-resolution training model corresponding to the original face image.

[0041] It should be understood that positive and negative samples are generated in order to lay the groundwork for comparative learning.

[0042] It should be understood that features extracted from the images are used for contrastive learning to enhance the quality of the generated super-resolution images.

[0043] Specifically, generating positive and negative samples, the face SR task aims to transform low-resolution face images into clear, realistic, and high-resolution images. Since LR images are formed from image degradation processes and contain only low-frequency information, the SR model focuses on learning inverse translation to recover lost high-frequency components (e.g., edge and texture information). With this in mind, effective and task-specific data augmentation can be employed to generate positive samples (i.e., the positive sample set) and negative samples (i.e., the negative sample set).

[0044] Specifically, the super-resolution model to be trained can be the DIDnet network, which consists of an upsampling block (G sub-network) and a downsampling block (F sub-network). Sub-network G contains two basic upsampling blocks, and sub-network F is designed to generate a downsampled LR image from the HR image, which is composed of the two downsampling blocks.

[0045] In the above embodiments, the super-resolution training model is obtained by sequentially training and analyzing the super-resolution images of the super-resolution training model with the original face images. This can make the super-resolution images go away from the smooth results, enabling the network to extract more detailed information, obtain information-rich positive and negative pairs for the super-resolution images, and maximize the identity similarity between the super-resolution face and the real face in the feature space.

[0046] Optionally, as an embodiment of the present invention, the upsampling block includes a plurality of sequentially arranged residual channel attention blocks, a first 3×3 convolutional layer, a pixel recombination layer, and a 1×1 convolutional layer;

[0047] The process of step S11 includes:

[0048] Each original face image is magnified sequentially by multiple residual channel attention blocks to obtain magnified face images corresponding to each original face image;

[0049] The first 3×3 convolutional layer sequentially performs initial feature extraction on each of the magnified face images to obtain the face image after initial feature extraction corresponding to each of the original face images;

[0050] The pixel recombination layer sequentially upsamples each of the face images after the initial feature extraction to obtain a face image to be extracted corresponding to each of the original face images.

[0051] The 1×1 convolutional layer sequentially extracts features from each of the face images to be feature extracted, resulting in upsampled face images corresponding to each of the original face images.

[0052] Preferably, the number of residual channel attention blocks can be 30.

[0053] Specifically, the residual channel attention block integrates CA and residual concepts, and the specific steps are as follows:

[0054] Given a feature input, a ReLU convolution operation is first performed to obtain f. Then, f is rescaled using a CA module to obtain x. Finally, x is added to the input to obtain the output feature. All convolution operations use 3×3 kernels.

[0055] The specific steps of the CA module are as follows:

[0056] The input is an H×W×C feature. First, a spatial global average pooling is performed to obtain a 1×1×C channel description. Then, a downsampling layer and an upsampling layer are used to obtain the weight coefficients of each channel. Multiplying the weight coefficients with the original feature yields the scaled new feature. The whole process is actually a reweighting of the features of different channels.

[0057] Both downsampling and upsampling layers utilize 1×1 convolutions. The downsampling layer reduces the number of channels by a factor of r and uses ReLU activation, while the upsampling layer uses Sigmoid activation. The number of channels used is C = 64, and r = 16.

[0058] Specifically, each upsampling block consists of three parts: a main module, an upsampling module, and a 1×1 convolutional layer. Furthermore, each main module comprises 30 residual channel attention blocks (RCABs), and each upsampling module consists of a 3×3 convolutional layer (i.e., the first 3×3 convolutional layer) and a pixel reconstruction layer, used for upsampling the feature map. The pixel reconstruction layer uses PixelShuffle, an upsampling method that can enlarge the scaled-down feature map by a corresponding factor, replacing interpolation or deconvolution.

[0059] In the above embodiments, the upsampled face image is obtained by sequentially upsampling and analyzing each original face image through upsampling blocks. This can replace interpolation or deconvolution, and at the same time, it extracts more accurate feature information, which can make the super-resolution image go far away from the smooth result, and enable the network to extract more detailed information.

[0060] Optionally, as an embodiment of the present invention, the downsampling block includes a second 3×3 convolutional layer, a LeakyReLU activation function layer, and a third 3×3 convolutional layer;

[0061] The process of step S12 includes:

[0062] The second 3×3 convolutional layer sequentially performs a third feature extraction on each of the upsampled face images to obtain a face image with the third feature extraction corresponding to each of the original face images;

[0063] The LeakyReLU activation function layer sequentially performs the first mapping process on each of the third feature extraction face images to obtain the mapped face images corresponding to each of the original face images;

[0064] The third 3×3 convolutional layer sequentially performs a fourth feature extraction on each of the mapped face images to obtain the original face super-resolution image corresponding to each of the original face images.

[0065] It should be understood that each downsampling block consists of repeated applications of a 3×3 convolutional layer (i.e., the second 3×3 convolutional layer), followed by the activation function LeakyReLU and a 3×3 convolutional layer (i.e., the third 3×3 convolutional layer).

[0066] In the above embodiments, the original face super-resolution image is obtained by sequentially downsampling and analyzing the original face image through downsampling blocks. This can generate a downsampled LR image from the HR image, extract more accurate feature information, and make the super-resolution image go away from the smoothed result, so that the network can extract more detailed information.

[0067] Optionally, as an embodiment of the present invention, step S13 includes the following process:

[0068] Image enhancement processing is performed sequentially on each of the original face super-resolution images using multiple preset filters and the first formula, resulting in a positive sample set corresponding to each of the original face images. The first formula is:

[0069]

[0070] Among them, I P Let i be the set of positive samples corresponding to the i-th original face super-resolution image. Let be the i-th original face super-resolution image, RF be the image enhancement operation, and sp be the image super-resolution image. i Let be the i-th positive sample image, and P be the number of positive sample images.

[0071] It should be understood that multiple sharpened images (i.e., positive sample images) are further generated by applying different high-pass filters (i.e., the preset filters) to the HR image (i.e., the original face super-resolution image), and all the sharpened images (i.e., the positive sample images) are used as the positive sample set as positive set I. P For the i-th image, its positive sample set is represented as follows:

[0072]

[0073] Where P is the number of positive samples. RF is the image augmentation operation. To generate more positive samples, different high-pass filters are applied to the HR face image.

[0074] In the above embodiments, positive sample sets are obtained by sequentially performing image enhancement processing on each original face super-resolution image through multiple preset filters and the first formula, enabling the network to extract more detailed information and obtain information-rich positive and negative pairs for the super-resolution image.

[0075] Optionally, as an embodiment of the present invention, step S14 includes the following process:

[0076] The original super-resolution face images are sequentially degraded using multiple preset filters and the second equation to obtain a negative sample set corresponding to each original face image. The second equation is:

[0077]

[0078] Among them, I N Let j be the negative sample set corresponding to the j-th original face super-resolution image. Let be the j-th original face super-resolution image, DG be the degradation function, and sn be the face super-resolution image. j Let N be the j-th negative sample image, and N be the number of negative sample images.

[0079] It should be understood that, inspired by negative sample mining and adversarial training methods, face images that are very similar to real face images can be input into the contrastive learning model.

[0080] Specifically, slightly blurred images generated from real face images are used as the negative sample set I. N By using multiple negative samples to narrow the solution space, the performance of SR networks with contrastive loss can be further improved.

[0081] This makes the reconstructed SR face image closer to a real face. For the i-th image, its negative sample set is represented as follows:

[0082]

[0083] Where N is the number of negative samples, and DG is the degradation function that processes HR face images using a random Gaussian kernel.

[0084] In the above embodiments, negative sample sets are obtained by sequentially degrading each original face super-resolution image through multiple preset filters and the second equation, which reduces the resolution space and can further improve the performance of the SR network with contrast loss, thereby making the reconstructed SR face image closer to the real face.

[0085] Optionally, as an embodiment of the present invention, the process of step S15 includes:

[0086] Each of the original super-resolution face images and the positive sample set corresponding to each of the original face images are sequentially combined as the corresponding positive image set for each of the original face images;

[0087] Each original super-resolution face image and the negative sample set corresponding to each original face image are sequentially combined into a negative pair image set corresponding to each original face image.

[0088] The feature extraction network is used to extract feature maps sequentially from each of the positive image sets and the negative image sets corresponding to each of the original face images, so as to obtain the positive feature maps and the negative feature maps corresponding to each of the original face images.

[0089] The contrastive learning loss value corresponding to each original face image is obtained by sequentially calculating the contrastive learning loss value using the third formula on each of the positive feature maps, the original face super-resolution image corresponding to each of the original face images, the positive sample set corresponding to each of the original face images, the negative feature maps corresponding to each of the original face images, and the negative sample set corresponding to each of the original face images. The third formula is:

[0090]

[0091] Among them, L c To compare the learning loss values, (G(I) SR ), G(I p )) represents the feature map, (G(I) SR ), G(I N )) is the negative pair feature map, I SR For the original face super-resolution image, I p For the positive sample set, I N Let E be the negative sample set, E() be the expected value, and α and β be hyperparameters.

[0092] It should be understood that positive pairs (i.e., the set of positive image pairs) and negative pairs (i.e., the set of negative image pairs) are constructed, and the latent feature spaces of these image pairs are compared.

[0093] It should be understood that the goal of contrastive learning is to learn a representation that pulls positive pairs together in the latent feature space and separates the representations between negative pairs.

[0094] It should be understood that the feature map is a matrix.

[0095] It should be understood that G() represents feature extraction (i.e., the extraction of the feature map).

[0096] Specifically, the pair (i.e., the pair image set) consists of super-resolution images I. SR (i.e., the original super-resolution face image) and enhanced image I P (i.e., the positive sample set). Similarly, the negative pairs are composed of the super-resolution image I. SR (i.e., the original super-resolution face image) and the degraded image I N (i.e., the negative sample set) constitutes the set.

[0097] It should be understood that the feature extraction network is used to extract features from an image. The shallow layers of the network typically extract lower-level features, such as image texture and borders.

[0098] Specifically, for the latent feature space, IECL-FSR uses a feature extraction network to extract feature maps f∈R. C×H×W ,in It is the inner product of feature maps i and j in layer (i, j).

[0099]

[0100] Where k represents the vector length, and then a set of Gram matrices {G} is obtained from layers 1, ..., L of the feature extraction network. 1 G 2 , ..., G L The Gram matrix G is a quantitative description of the latent image features. Similar to triplet loss, the objective is to minimize I. SR and I P The distance between them, both having the same identity, and maximizing I SR And I with different identities N Distance between:

[0101] d(G(I SR ), G(I P ))<<d(G(I SR ), G(I N ))

[0102] d(E(I SR ), E(I P ))<<d(E(I SR ), E(I N ))

[0103] Where E represents the expected value. Hope I SR and I P The distance between features is less than I SR and I NDistance between features. Contrastive learning loss L c (i.e., the contrastive learning loss value) can be expressed as:

[0104] L c =max{d(G(I SR ), G(I P ))-d(G(I SR ), G(I N ))+α,0}

[0105] +max{d(E(I SR ), E(I P ))-d(E(I SR ), E(I N ))+β,0}

[0106] Where α and β are hyperparameters (i.e., margins in triplet loss), the above loss function is formulated using triplet loss in this implementation.

[0107] In the above embodiments, the contrastive learning loss values ​​are analyzed sequentially for each original face super-resolution image, positive sample set, and negative sample set to obtain the contrastive learning loss value corresponding to each original face image.

[0108] Optionally, as an embodiment of the present invention, the process of training and analyzing the face reconstruction images of the face reconstruction training model by using all original face images and all target super-resolution images to obtain the face reconstruction training model includes:

[0109] S31: Extract feature vectors from each of the original face images and each of the target super-resolution images using a deep convolutional neural network to obtain original face feature vectors corresponding to each of the original face images and target super-resolution image feature vectors corresponding to each of the target super-resolution images.

[0110] S32: The L2 regularization algorithm is used to perform regularization processing on the original face feature vector corresponding to each of the original face images and the target super-resolution image feature vector corresponding to each of the target super-resolution images, respectively, to obtain the regularized face image corresponding to each of the original face images and the regularized super-resolution image corresponding to each of the original face images.

[0111] S33: The identity loss value is obtained by calculating the identity loss value for all regularized face images and all regularized super-resolution images using the fourth formula. The fourth formula is:

[0112]

[0113] Among them, L idt This represents the value of identity loss. Let i be the regularized face image corresponding to the i-th original face image. Let N' be the regularized super-resolution image corresponding to the i-th original face image, and N' be the number of original face images.

[0114] S34: Determine whether the identity loss value is greater than the preset second loss threshold. If yes, update the parameters of the face reconstruction training model according to the identity loss value, and use the updated face reconstruction training model as the face reconstruction training model, and return to step S31; if no, use the face reconstruction training model as the face reconstruction training model.

[0115] It should be understood that obtaining a face reconstruction training model can better preserve the identity of the super-resolution image and improve the reconstruction effect.

[0116] It should be understood that, based on contrastive learning, the identity feature representation of the face recognition network is used as a supervisory signal, and the identity features are used to further constrain the results of contrastive learning in order to obtain higher quality face reconstruction images.

[0117] It should be understood that regularization, in linear algebra theory, refers to the process where ill-posed problems are typically defined by a set of linear algebraic equations, and this set of equations often originates from ill-posed inverse problems with large condition numbers. Large condition numbers mean that rounding errors or other errors can severely affect the results of the problem.

[0118] It should be understood that linear models are often used to handle regression and classification tasks. To prevent the model from overfitting, L1 and L2 regularization are used to reduce the model's complexity. Many papers on linear regression model regularization mention that L1 reduces complexity by sparse parameters (reducing the number of parameters), while L2 (i.e., the L2 regularization algorithm) reduces complexity by decreasing the size of the parameter values. There are many regularization methods, and L2 regularization is one of the common ones. Its implementation mainly involves two steps. The first step is to add a "tail" to the cost function; the second step is to add a "tail" when calculating the partial derivatives.

[0119] Specifically, FaceNet, pre-trained on VGGFace2, maps the image (i.e., the original face image or the target super-resolution image) into a 128-dimensional feature space through the deep convolutional neural network to obtain the corresponding 128-dimensional feature vector (i.e., the original face feature vector or the target super-resolution image feature vector). Then, L2 regularization is applied to the feature vector (i.e., the original face feature vector or the target super-resolution image feature vector) to filter out effective features. Then, through identity-level constraints, a FaceNet model with better performance can be obtained. By using the identity features extracted by FaceNet and applying them to IECL-FSR, the generation of reconstructed images can be constrained, maintaining facial identity consistency between low-resolution and high-resolution face images. LR This represents a low-resolution face image. I represents the reconstructed face image. HR If we represent a high-resolution face image of the target, then the identity loss can be expressed as:

[0120]

[0121] Reconstructed high-resolution face image I SR =F SR (I LR ) and original high-resolution face image I HR Projected onto the identity feature space, (i.e., the regularized face image) and (That is, the regularized super-resolution image) is the identity feature extracted from the fully connected layer of the face recognition model.

[0122] In the above embodiments, the face reconstruction training model is obtained by training and analyzing the face reconstruction images of the face reconstruction training model through all the original face images and all the target super-resolution images. This can better preserve the identity of the super-resolution images and improve the reconstruction effect of the images.

[0123] Optionally, as another embodiment of the present invention, the present invention aims to reconstruct high-fidelity and identity-aware HR face images. The algorithm utilizes contrastive learning to generate negative samples by blurring the real face image, thus moving the super-resolution image away from the smoothed result, and to sharpen the real image to generate information-rich positive samples. This allows the network to extract more detailed information, obtaining information-rich positive-negative pairs for the super-resolution image. Furthermore, by embedding positive, negative, and anchor samples into a feature space that effectively defines the contrastive loss, the anchor samples are moved away from negative samples and closer to positive samples. Therefore, the super-resolution image can be well separated from negative samples while remaining close to positive samples, thereby increasing the similarity between anchor samples and positive samples and maximizing the identity similarity between the super-reconstructed face and the real face in the feature space. In addition, the identity attribute features obtained using a pre-trained model are combined to jointly optimize the reconstructed image to improve its visual fidelity.

[0124] Optionally, as another embodiment of the present invention, the present invention uses three indicators to quantitatively analyze the performance of SR images, including Peak Signal-to-Noise Ratio (PSNR), Structural Similarity (SSIM), and Learned Perceptual Image Path Similarity (LPIPS). Higher values ​​for PSNR and SSIM indicate better generated results, while lower values ​​for LPIPS indicate better generated results.

[0125] LPIPS is used to measure the difference between two images and is more in line with human perception than PSNR and SSIM. Its definition is as follows:

[0126]

[0127] d is the distance between x0 and x. The lower the LPIPS value, the more similar the two images are.

[0128] To verify the practicality of the present invention, it was compared with the most advanced methods currently available. The comparison methods included general image super-resolution algorithms, face image super-resolution algorithms, and image super-resolution algorithms based on contrastive learning. General image super-resolution algorithms included SRFBN and RCAN, while face image super-resolution algorithms included SISN and the default DIDnet algorithm.

[0129] The quantitative experimental results are shown in Table 1.

[0130] Table 1 shows the PSNR, SSIM, and LPIPS values ​​of different super-resolution algorithms on the FFHQ dataset.

[0131]

[0132] Table 1 lists the experimental results of different state-of-the-art methods and the proposed method (IECL-FSR) on the FFHQ test dataset under a scaling factor of ×4. Bold numbers indicate the best results. It can be seen that IECL-FSR outperforms existing methods in all three metrics. Specifically, IECL-FSR achieves optimal performance by combining a contrastive learning framework and incorporating identity attribute features.

[0133] Compared to other methods, this invention offers superior reconstruction capabilities and excellent reconstruction performance. By combining contrastive learning and recurrent networks, and incorporating identity features, the reconstruction results contain clearer and richer textures. Furthermore, the use of feature-preserving loss allows for better retention of global features, achieving optimal visual effects.

[0134] Figure 2 This is a block diagram of a face image reconstruction device provided in an embodiment of the present invention.

[0135] Alternatively, as another embodiment of the present invention, such as Figure 2 As shown, a face image reconstruction device includes:

[0136] The first training and analysis module is used to import multiple original face images and construct a super-resolution model to be trained. The super-resolution model to be trained is then trained and analyzed using each of the original face images to obtain a super-resolution training model corresponding to each of the original face images.

[0137] The image recognition module is used to perform image recognition on each of the original face images by using a super-resolution training model corresponding to each of the original face images, so as to obtain a target super-resolution image corresponding to each of the original face images;

[0138] The second training and analysis module is used to construct a face reconstruction training model. It performs face reconstruction image training analysis on the face reconstruction training model using all original face images and all target super-resolution images to obtain a face reconstruction training model.

[0139] The face image reconstruction result module is used to reconstruct each of the target super-resolution images using the face reconstruction training model, to obtain face reconstruction images corresponding to each of the original face images, and to use all the face reconstruction images as the face image reconstruction result.

[0140] Optionally, another embodiment of the present invention provides a face image reconstruction apparatus, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements the face image reconstruction method described above. This apparatus may be a computer or similar device.

[0141] Optionally, another embodiment of the present invention provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the face image reconstruction method as described above.

[0142] It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article, or apparatus.

[0143] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working process of the above-described apparatus and unit can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0144] In the several embodiments provided in this application, it should be understood that the disclosed apparatus and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative. For instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed.

[0145] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of the embodiments of the present invention, depending on actual needs.

[0146] Furthermore, the functional units in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0147] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this invention, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0148] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A method for reconstructing a face image, characterized in that, Includes the following steps: Import multiple original face images and construct a super-resolution training model. Then, use each of the original face images to sequentially perform super-resolution image training analysis on the super-resolution training model to obtain a super-resolution training model corresponding to each of the original face images. By performing image recognition on each of the original face images using a super-resolution training model corresponding to each of the original face images, a target super-resolution image corresponding to each of the original face images is obtained. A face reconstruction training model is constructed by performing face reconstruction image training analysis on the face reconstruction training model using all original face images and all target super-resolution images to obtain a face reconstruction training model. The face reconstruction training model is used to reconstruct each of the target super-resolution images to obtain face reconstruction images corresponding to each of the original face images, and all face reconstruction images are used as the face image reconstruction result. The super-resolution training model includes a pair of sequentially arranged upsampling blocks and a pair of sequentially arranged downsampling blocks. The process of sequentially performing super-resolution image training analysis on the super-resolution training model using each of the original face images to obtain a super-resolution training model corresponding to each of the original face images includes: S11: The upsampling block sequentially performs upsampling analysis on each of the original face images to obtain upsampled face images corresponding to each of the original face images; S12: The downsampling block is used to perform downsampling analysis on each of the original face images in sequence to obtain the original face super-resolution image corresponding to each of the original face images; S13: Image enhancement processing is performed on each of the original face super-resolution images sequentially through multiple preset filters to obtain a positive sample set corresponding to each of the original face images; S14: The original super-resolution face images are degraded sequentially using multiple preset filters to obtain a negative sample set corresponding to each original face image; S15: Analyze the contrastive learning loss values ​​of each of the original face super-resolution images, the positive sample set corresponding to the original face image, and the negative sample set corresponding to the original face image in sequence to obtain the contrastive learning loss value corresponding to each of the original face images. S16: Sequentially determine whether each of the contrastive learning loss values ​​is greater than a preset first loss threshold. If so, update the parameters of the super-resolution training model according to the contrastive learning loss value, and use the updated super-resolution training model as the super-resolution training model corresponding to the original face image, and return to step S11. If not, then the super-resolution model to be trained will be used as the super-resolution training model corresponding to the original face image.

2. The face image reconstruction method according to claim 1, characterized in that, The upsampling block includes multiple sequentially arranged residual channel attention blocks, and a first... Convolutional layers, pixel recombination layers and Convolutional layer; The process of step S11 includes: Each of the original face images is magnified sequentially by multiple residual channel attention blocks to obtain magnified face images corresponding to each of the original face images; Through the first The convolutional layer sequentially performs initial feature extraction on each of the magnified face images to obtain the face image with initial feature extraction corresponding to each of the original face images; The pixel recombination layer sequentially upsamples each of the face images after the initial feature extraction to obtain a face image to be extracted corresponding to each of the original face images. Through the above The convolutional layer sequentially performs feature extraction on each of the face images to be feature extracted, resulting in upsampled face images corresponding to each of the original face images.

3. The face image reconstruction method according to claim 1, characterized in that, The downsampling block includes a second Convolutional layers, LeakyReLU activation function layers, and the third Convolutional layer; The process of step S12 includes: Through the second The convolutional layer sequentially performs a third feature extraction on each of the upsampled face images to obtain a third feature-extracted face image corresponding to each of the original face images; The LeakyReLU activation function layer sequentially performs the first mapping process on each of the third feature extraction face images to obtain the mapped face images corresponding to each of the original face images; Through the third The convolutional layer sequentially performs a fourth feature extraction on each of the mapped face images to obtain the original face super-resolution image corresponding to each of the original face images.

4. The face image reconstruction method according to claim 1, characterized in that, The process of step S13 includes: Image enhancement processing is performed sequentially on each of the original face super-resolution images using multiple preset filters and the first formula, resulting in a positive sample set corresponding to each of the original face images. The first formula is: ， in, For the first A set of positive samples corresponding to the original super-resolution face images. For the first One original super-resolution image of a human face. RF For image enhancement operations, For the first One positive sample image, P This represents the number of positive sample images.

5. The face image reconstruction method according to claim 1, characterized in that, The process of step S14 includes: The original super-resolution face images are sequentially degraded using multiple preset filters and the second equation to obtain a negative sample set corresponding to each original face image. The second equation is: ， in, For the first A negative sample set corresponding to each original super-resolution face image. For the first One original super-resolution image of a human face. It is a degrading function. For the first One negative sample image, N This represents the number of negative sample images.

6. The face image reconstruction method according to claim 1, characterized in that, The process of step S15 includes: Each of the original super-resolution face images and the positive sample set corresponding to each of the original face images are sequentially combined as the corresponding positive image set for each of the original face images; Each original super-resolution face image and the negative sample set corresponding to each original face image are sequentially combined into a negative pair image set corresponding to each original face image. The feature extraction network is used to extract feature maps sequentially from each of the positive image sets and the negative image sets corresponding to each of the original face images, so as to obtain the positive feature maps and the negative feature maps corresponding to each of the original face images. The contrastive learning loss value corresponding to each original face image is obtained by sequentially calculating the contrastive learning loss value using the third formula on each of the positive feature maps, the original face super-resolution image corresponding to each of the original face images, the positive sample set corresponding to each of the original face images, the negative feature maps corresponding to each of the original face images, and the negative sample set corresponding to each of the original face images. The third formula is: ， in, To compare the learning loss values, To face the feature map, For negative pair feature maps, The original super-resolution image of the face. For positive sample sets, For negative sample set, As the expected value, and All of these are hyperparameters.

7. The face image reconstruction method according to claim 1, characterized in that, The process of training and analyzing the face reconstruction images of the face reconstruction training model using all original face images and all target super-resolution images to obtain the face reconstruction training model includes: S31: Extract feature vectors from each of the original face images and each of the target super-resolution images using a deep convolutional neural network to obtain original face feature vectors corresponding to each of the original face images and target super-resolution image feature vectors corresponding to each of the target super-resolution images. S32: The L2 regularization algorithm is used to perform regularization processing on the original face feature vector corresponding to each of the original face images and the target super-resolution image feature vector corresponding to each of the target super-resolution images, respectively, to obtain the regularized face image corresponding to each of the original face images and the regularized super-resolution image corresponding to each of the original face images. S33: The identity loss value is obtained by calculating the identity loss value for all regularized face images and all regularized super-resolution images using the fourth formula. The fourth formula is: ， in, This represents the value of identity loss. For the first The regularized face image corresponding to the original face image For the first The regularized super-resolution image corresponding to the original face image. The number of original face images; S34: Determine whether the identity loss value is greater than the preset second loss threshold. If yes, update the parameters of the face reconstruction training model according to the identity loss value, and use the updated face reconstruction training model as the face reconstruction training model, and return to step S31; if no, use the face reconstruction training model as the face reconstruction training model.

8. A face image reconstruction device, characterized in that, include: The first training and analysis module is used to import multiple original face images and construct a super-resolution model to be trained. The super-resolution model to be trained is then trained and analyzed using each of the original face images to obtain a super-resolution training model corresponding to each of the original face images. The image recognition module is used to perform image recognition on each of the original face images by using a super-resolution training model corresponding to each of the original face images, so as to obtain a target super-resolution image corresponding to each of the original face images; The second training and analysis module is used to construct a face reconstruction training model. It performs face reconstruction image training analysis on the face reconstruction training model using all original face images and all target super-resolution images to obtain a face reconstruction training model. The face image reconstruction result module is used to reconstruct each of the target super-resolution images using the face reconstruction training model, to obtain face reconstruction images corresponding to each of the original face images, and to use all the face reconstruction images as the face image reconstruction result. The super-resolution model to be trained includes a pair of upsampled blocks arranged in sequence and a pair of downsampled blocks arranged in sequence. The first training analysis module is specifically used for: S11: The upsampling block sequentially performs upsampling analysis on each of the original face images to obtain upsampled face images corresponding to each of the original face images; S12: The downsampling block is used to perform downsampling analysis on each of the original face images in sequence to obtain the original face super-resolution image corresponding to each of the original face images; S13: Image enhancement processing is performed on each of the original face super-resolution images sequentially through multiple preset filters to obtain a positive sample set corresponding to each of the original face images; S14: The original super-resolution face images are degraded sequentially using multiple preset filters to obtain a negative sample set corresponding to each original face image; S15: Analyze the contrastive learning loss values ​​of each of the original face super-resolution images, the positive sample set corresponding to the original face image, and the negative sample set corresponding to the original face image in sequence to obtain the contrastive learning loss value corresponding to each of the original face images. S16: Sequentially determine whether each of the contrastive learning loss values ​​is greater than a preset first loss threshold. If so, update the parameters of the super-resolution training model according to the contrastive learning loss value, and use the updated super-resolution training model as the super-resolution training model corresponding to the original face image, and return to step S11. If not, then the super-resolution model to be trained will be used as the super-resolution training model corresponding to the original face image.

9. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by a processor, it implements the face image reconstruction method as described in any one of claims 1 to 7.

Images