Method, system and storage medium for reconstructing super-resolution face images from low-definition images

Abstract

The present invention discloses a method, system and storage medium for reconstructing a super-resolution face image from a low-definition image, including: inputting a low-resolution image into a pre-trained progressive generative adversarial network to initialize hidden codes and channel weight parameters; inputting hidden codes and channel weights into the iteration unit of the pre-trained progressive generative adversarial network to obtain iterative multiple hidden codes and channel weights; judging whether the number of iterations of the multiple hidden codes and channel weights is greater than a preset iteration number threshold, and generating a final super-resolution high-definition face image based on the output multiple hidden codes and channel weights. The present invention groups multiple hidden codes and iterates and optimizes them under different sub-network structures of the pre-trained progressive generative adversarial network, so that multiple latent codes learn features at different levels, improve the realism of the super-resolution face image, and thus reconstruct the super-resolution face image.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence, and in particular to a method, system and storage medium for reconstructing a super-resolution face image from a low-definition image. Background Art

[0002] In recent years, with the continuous development of the information age, people are increasingly relying on the Internet to obtain information. According to statistics, 75% of the information a person obtains comes from vision, but the amount of data after the digitization of high-definition images is very large. When transmitting large-resolution face images on the Internet, in order to save network bandwidth and speed, the face images obtained from the Internet are low-resolution images after compressed transmission, which seriously affects the user's acquisition of facial features, face shape and other detailed information of the face image. Due to the particularity of the face, it is often difficult to restore the details of the face after compression, especially for large-scale face super-resolution, the realism and identity information of the face will be distorted. Face super-resolution can bring about improvements in speed, security and quality in face image compression, which has attracted more and more attention from academia and the business community.

[0003] Compared with high-definition face images, low-definition face images often have blurred backgrounds, uneven lighting, and contain more noise. In addition, a lot of detailed facial information is lost in low-definition face features. Super-resolution images often lose many facial details and even the original facial identity features, affecting the quality perception. Directly using conventional face super-resolution algorithms often results in poor results, which greatly affects the performance of the face super-resolution system.

[0004] Generally speaking, solutions to the problem of super-resolution of low-definition faces can be divided into two categories. One is the supervised method, which often requires a large number of face image pairs, each pair of images contains a high-definition face and a corresponding low-definition face, but in reality it is difficult to obtain real image pairs, most of which are low-definition images generated by artificial downsampling, which will make the learned model unable to adapt well to the image degradation problem in real scenes. The other is the unsupervised method, which uses the pre-trained model as prior knowledge to guide the generation of low-definition images and uses the optimization algorithm to optimize the input of the generative adversarial network. It is not applicable in ultra-low-definition situations, and lacks facial details, which makes the model unable to extract identification features. Therefore, the results obtained by this type of method are still not very ideal. Summary of the invention

[0005] Purpose of the invention: In order to overcome the deficiencies in the prior art, the present invention provides a method, system and storage medium for reconstructing super-resolution face images from low-definition images; through the present invention, multiple latent codes are grouped, and they are iteratively optimized under different sub-network structures of a pre-trained progressive generative adversarial network, so that multiple latent codes learn features at different levels, thereby improving the realism of super-resolution face images, introducing the identity preservation loss of the face recognition network, constraining the identity features to remain unchanged as much as possible, introducing the decoupling of intermediate features corresponding to multiple latent codes, so that each latent code is semantically decoupled, and using the pre-trained progressive generative adversarial network and the face recognition network to reconstruct the super-resolution face image.

[0006] Technical solution: In the first aspect, the present invention provides a method for reconstructing a super-resolution face image from a low-definition image, comprising:

[0007] Input the low-resolution image into the pre-trained progressive generative adversarial network to obtain hidden codes and channel weights; the parameter weights of the pre-trained progressive generative adversarial network are trained by a large amount of high-definition face data;

[0008] Input the hidden code and channel weight into the iterative unit of the pre-trained progressive generative adversarial network to obtain iterative multiple hidden codes and channel weights;

[0009] Determine whether the number of iterations of the multiple hidden codes and channel weights is greater than a preset iteration number threshold, and output the multiple hidden codes and channel weights that are greater than the preset iteration number threshold;

[0010] The final super-resolution high-definition face image is generated based on the output multiple hidden codes and channel weights.

[0011] In a further embodiment, the pre-trained progressive generative adversarial network is preset with an iteration number threshold and a Gaussian random initialization parameter.

[0012] In a further embodiment, the method of inputting the hidden codes and channel weights into the iterative unit of the pre-trained progressive generative adversarial network to obtain iterative multiple hidden codes and channel weights includes:

[0013] The hidden codes and the corresponding adaptive channel weight parameters are divided into two groups of parameters, and the two groups of parameters are respectively substituted into the pre-trained progressive generative adversarial network iteration unit to obtain iterative multiple hidden codes and channel weights. Among them, the pre-trained progressive generative adversarial network iteration unit performs Gaussian random initialization on the parameters of the hidden codes and mean initialization on the parameters of the channel weights in the first iteration;

[0014] The iterative unit is composed of iterative calculation units in multiple sub-networks associated with different combined layers of a pre-trained progressive generative adversarial network, and optimizes hidden codes and adaptive channel weights through back propagation of multiple loss functions during the iterative process to obtain multiple hidden codes and channel weights that are iteratively optimized multiple times.

[0015] In a further embodiment, the two groups of parameters are respectively a first group of parameters and a second group of parameters; the combined layer of the pre-trained progressive generative adversarial network includes: a first combined layer, a second combined layer and a third combined layer;

[0016] In a further embodiment, the method of dividing the latent codes and the corresponding adaptive channel weight parameters into two groups of parameters, and substituting the two groups of parameters into the pre-trained progressive generative adversarial network iteration unit to obtain iterative multiple latent codes and channel weights includes:

[0017] The first group of parameters is iterated on the first subnetwork and the second subnetwork of the first combination layer, and the second group of parameters is iterated on the first subnetwork and the second subnetwork of the second combination layer, and two groups of latent codes and weight parameters are output; wherein each group of latent codes is input to the first subnetwork before the corresponding combination layer, each latent code generates a corresponding first intermediate feature through the first subnetwork, and the first intermediate features generated by each group of latent codes are combined into second intermediate features according to the adaptive channel weights, and then all the second intermediate features are accumulated and fused to generate the third intermediate features; the third intermediate features are input to the second subnetwork to generate a face image, and the latent codes and adaptive channel weights are optimized by back-propagation through multiple loss functions, respectively, to obtain two groups of latent codes and adaptive channel weights after iterative optimization;

[0018] The two sets of hidden codes and weight parameters are iterated through the first sub-network and the second sub-network of the third combination layer respectively, and multiple hidden codes and adaptive channel weights are output.

[0019] In a further embodiment, determining whether the number of iterations of the multiple hidden codes and the channel weights is greater than a preset iteration number threshold further includes:

[0020] For multiple hidden codes and adaptive channel weights whose iteration number is less than a preset iteration number threshold, re-input the iteration unit to update until the iteration number is greater than the preset iteration number threshold, and then output the updated multiple hidden codes and adaptive channel weights;

[0021] The final super-resolution high-definition face image is obtained by super-resolution calculation based on the updated multiple hidden codes and adaptive channel weights;

[0022] Among them, there are multiple preset iteration number thresholds, which respectively control the output and iteration loop of the iterative calculation units in multiple sub-networks in turns; the multiple conditional expressions for controlling the output of the iterative calculation units in multiple sub-networks in turns are as follows:

[0023]

[0024] In the formula, l represents the current iteration round, L represents the total number of iterations, k1 and k2 respectively represent the number of rounds for iterative optimization in the combined layers of the l1-th and l2-th layers of the GAN, and l n represents the index position of the intermediate layer of the network and the index l of the intermediate layer of the network n satisfies the relationship of l1 < l2 < l3; among them, l1 represents the index position of the first combined layer, l2 represents the index position of the second combined layer, and l3 represents the index position of the third combined layer and is processed iteratively in sequence until the preset maximum number of iterations L is reached;

[0025] The expressions for the multiple latent encodings and adaptive channel weights after updating the iteration are as follows:

[0026]

[0027] In the formula, z (n) is the multiple latent encoding, α (n) is the adaptive channel weight, N is the total number of latent encodings, L is the total loss function, X Inv is the high-definition face image of the final super-resolution, and X LR is the low-resolution image.,

[0028] In a further embodiment, the multiple loss functions include: a first loss function, a second loss function, a third loss function, and a fourth loss function;

[0029] Among them, the first loss function is a difference calculation function for the first intermediate feature and the second intermediate feature of the first sub-network;

[0030] The second loss function is the mean square error loss;

[0031] The third loss is the perceptual loss;

[0032] The fourth loss function is a cosine similarity calculation function for the identity vector of the generated reconstructed image and the identity vector of the low-resolution image;

[0033] Among them, the expression of the first loss function is:

[0034]

[0035] In the formula, i is the x coordinate of the feature map, j is the y coordinate of the feature map, Represents the second intermediate feature obtained by weighting the first intermediate feature generated by each hidden code on the x-coordinate of the feature map, The second intermediate feature obtained by weighting the first intermediate feature on the y coordinate of the feature map;

[0036] The expression of the second loss function is:

[0037] L pix =|down(x inv )-x LR |

[0038] Where, X inv represents the super-resolution face image generated in each iteration, and down() represents the downsampling operation on the face image;

[0039] The expression of the third loss function is:

[0040]

[0041] In the formula, Represents the network intermediate layer feature map output by the ultra-deep convolutional network VGG for large-scale image recognition.

[0042] The expression of the fourth loss function is:

[0043]

[0044] Where R represents the face recognition network trained with the additional angular margin loss, which calculates the cosine similarity between the identity vector and the identity vector of the low-resolution image, G(z i ) represents z i The reconstructed face image is input into the progressive generative adversarial network.

[0045] In a further embodiment, the method further includes adjusting the iteration number threshold based on the first loss function, the second loss function, the third loss function, and the fourth loss function. The preset iteration number threshold calculation formula is:

[0046] L tot =λ1L pix +λ2L div +λ3L f +λ4L id

[0047] Where λ1, λ2, λ3, and λ4 are proportional factors respectively.

[0048] In a second aspect, the present invention provides a system for reconstructing a super-resolution face image from a low-definition image, including a processor and a storage medium;

[0049] The storage medium is used to store instructions;

[0050] The processor is used to operate according to the instructions to execute the steps of the above method.

[0051] In a third aspect, the present invention provides a computer-readable storage medium having a computer program stored thereon, which implements the steps of the above method when executed by a processor.

[0052] Beneficial effects: Compared with the prior art, the present invention has the following advantages:

[0053] 1. Taking advantage of the different knowledge representations expressed by different intermediate layers in the pre-trained generative adversarial network, the features generated by the shallow network show high-level semantic features such as identity and color, and the features generated by the deep network show low-level semantic features such as structure and edge, and the multiple latent codes are grouped and iteratively optimized. The multiple latent codes are divided into two groups and placed in different intermediate component layers of the pre-trained network for iterative optimization, so that the two parts of the multiple latent codes learn the features of different levels of semantics, and the generated super-resolution face images have a stronger sense of reality.

[0054] 2. In the iterative optimization process, each intermediate feature will be generated after the first sub-network of the pre-trained model, and the adaptive channel weights are used to synthesize these intermediate features into new intermediate features. Each intermediate feature is responsible for reconstructing different areas in the image. Therefore, the difference of these intermediate features is used as a constraint to make the multiple latent codes perform their respective functions better, achieve the effect of decoupling, and help to further enhance the realism of the face image.

[0055] 3. In order to ensure that the generated super-resolution face image is not only realistic but also maintains the identity features of the original face image, a face recognition network is introduced to calculate the feature cosine similarity between the input low-definition face and the face generated by each latent code, so as to constrain the multiple latent codes to not deviate from the identity domain of the low-definition face during iterative optimization in the latent space. BRIEF DESCRIPTION OF THE DRAWINGS

[0056] Figure 1 is a flow chart of a method for reconstructing a super-resolution face image from a low-definition image provided by an embodiment of the present invention;

[0057] Figure 2(a) , 2(b) Low-resolution face images;

[0058] FIG3(a) is a diagram showing the super-resolution result of FIG2(a) using the super-resolution method for image processing based on multi-code generative adversarial network prior disclosed in Method 1;

[0059] FIG3( b ) is a diagram showing the super-resolution result of FIG2( b ) using the super-resolution method for image processing based on multi-code generative adversarial network priors disclosed in Method 1;

[0060] FIG4( a ) is a super-resolution result of FIG2( a ) using the method of the present invention;

[0061] FIG. 4( b ) is a super-resolution result of FIG. 2( b ) using the method of the present invention. DETAILED DESCRIPTION

[0062] In order to more fully understand the technical content of the present invention, the technical solution of the present invention is further introduced and illustrated in conjunction with specific embodiments below, but is not limited thereto.

[0063] Embodiment 1:

[0064] like Figure 1 This embodiment provides a method for reconstructing a super-resolution face image from a low-definition image, comprising:

[0065] Input the low-resolution image into the pre-trained progressive generative adversarial network to obtain hidden codes and channel weights; the parameter weights of the pre-trained progressive generative adversarial network are trained by a large amount of high-definition face data;

[0066] Input the hidden code and channel weight into the iterative unit of the pre-trained progressive generative adversarial network to obtain iterative multiple hidden codes and channel weights;

[0067] Determine whether the number of iterations of the multiple hidden codes and channel weights is greater than a preset iteration number threshold, and output the multiple hidden codes and channel weights that are greater than the preset iteration number threshold;

[0068] The final super-resolution high-definition face image is generated based on the output multiple hidden codes and channel weights.

[0069] Furthermore, the pre-trained progressive generative adversarial network has a preset iteration threshold and Gaussian random initialization parameters.

[0070] Furthermore, the method of inputting the hidden codes and channel weights into the iterative unit of the pre-trained progressive generative adversarial network to obtain iterative multiple hidden codes and channel weights includes:

[0071] The hidden codes and the corresponding adaptive channel weight parameters are divided into two groups of parameters, and the two groups of parameters are respectively substituted into the pre-trained progressive generative adversarial network iteration unit to obtain iterative multiple hidden codes and channel weights. Among them, the pre-trained progressive generative adversarial network iteration unit performs Gaussian random initialization on the parameters of the hidden codes and mean initialization on the parameters of the channel weights in the first iteration;

[0072] The iterative unit is composed of iterative calculation units in multiple sub-networks associated with different combined layers of a pre-trained progressive generative adversarial network, and optimizes hidden codes and adaptive channel weights through back propagation of multiple loss functions during the iterative process to obtain multiple hidden codes and channel weights that are iteratively optimized multiple times.

[0073] Furthermore, the two groups of parameters are respectively a first group of parameters and a second group of parameters; the combined layer of the pre-trained progressive generative adversarial network includes: a first combined layer, a second combined layer and a third combined layer;

[0074] Furthermore, the method of dividing the latent codes and the corresponding adaptive channel weight parameters into two groups of parameters, and substituting the two groups of parameters into the pre-trained progressive generative adversarial network iteration unit to obtain iterative multiple latent codes and channel weights includes:

[0075] The first group of parameters is iterated on the first subnetwork and the second subnetwork of the first combination layer, and the second group of parameters is iterated on the first subnetwork and the second subnetwork of the second combination layer, and two groups of latent codes and weight parameters are output; wherein each group of latent codes is input to the first subnetwork before the corresponding combination layer, each latent code generates a corresponding first intermediate feature through the first subnetwork, and the first intermediate features generated by each group of latent codes are combined into second intermediate features according to the adaptive channel weights, and then all the second intermediate features are accumulated and fused to generate the third intermediate features; the third intermediate features are input to the second subnetwork to generate a face image, and the latent codes and adaptive channel weights are optimized by back-propagation through multiple loss functions, respectively, to obtain two groups of latent codes and adaptive channel weights after iterative optimization;

[0076] The two sets of hidden codes and weight parameters are iterated on the first sub-network and the second sub-network of the third combination layer respectively, and multiple hidden codes and adaptive channel weights are output. In a further embodiment, determining whether the number of iterations of the multiple hidden codes and channel weights is greater than a preset iteration number threshold also includes:

[0077] For multiple hidden codes and adaptive channel weights whose iteration times are less than a preset iteration times threshold, update the iterations to obtain multiple hidden codes and adaptive channel weights after the update iterations;

[0078] Further, determining whether the number of iterations of the multiple hidden codes and the channel weights is greater than a preset iteration number threshold also includes:

[0079] For multiple hidden codes and adaptive channel weights whose iteration number is less than a preset iteration number threshold, re-input the iteration unit to update until the iteration number is greater than the preset iteration number threshold, and then output the updated multiple hidden codes and adaptive channel weights;

[0080] The final super-resolution high-definition face image is obtained by super-resolution calculation based on the updated multiple hidden codes and adaptive channel weights;

[0081] There are multiple preset iteration number thresholds, which respectively control the iterative calculation unit outputs and iterative cycles in multiple sub-networks;

[0082] The expressions of multiple hidden codes and adaptive channel weights after updating iterations are as follows:

[0083]

[0084] In the formula, z (n) is the multiple hidden code, α (n) is the adaptive channel weight, N is the total number of hidden codes, L is the total loss function, X Inv is the final super-resolution high-definition face image, X LR For low-resolution images.

[0085] In this embodiment, in each round of iteration, each hidden code z (n) Input the first subnetwork of the pre-trained model Then generate the corresponding first intermediate feature n is the number corresponding to the hidden code. Through the adaptive channel weight z (n) These first intermediate features To generate the second intermediate feature The generation method is defined as Where i is the x-coordinate of the feature map, j is the y-coordinate of the feature map, and c is the channel coordinate. The difference is used as the first loss function, and then all the second intermediate features are accumulated and fused. Generate the third intermediate feature. Input the third intermediate feature to the second sub-network of the pre-trained model Generated super-resolution image X Inv ;

[0086] The final super-resolution image generated is X Inv The specific calculation is as follows:

[0087]

[0088] In the formula, l represents the current iteration round, L represents the total number of iterations, k1 and k2 represent the rounds of iterative optimization in the combination layer of the l1th and l2th layers of GAN, respectively. n Indicates the network intermediate layer index position and the network intermediate layer index l nSatisfy the relationship of l1 < l2 < l3; where, l1 represents the index position of the first combined layer, l2 represents the index position of the second combined layer, l3 represents the index position of the third combined layer, and iterative processing is performed in sequence until the preset maximum number of iterations L is reached;

[0089] Furthermore, the multiple loss functions include: the first loss function, the second loss function, the third loss function, and the fourth loss function;

[0090] Among them, the first loss function is a difference calculation function for the first intermediate feature and the second intermediate feature of the first sub-network;

[0091] The second loss function is a downsampling degradation function for the face image generated on the second sub-network;

[0092] The third loss function is a function for extracting the first feature of the low-resolution image by the pre-trained progressive generative adversarial network;

[0093] The fourth loss function is a cosine similarity calculation function for the identity vector in the pre-trained progressive generative adversarial network and the identity vector of the low-resolution image;

[0094] Among them, the expression of the first loss function is:

[0095]

[0096] In the formula, represents the second intermediate feature obtained by weighting the first intermediate feature generated by each latent code on the x coordinate of the feature map, represents the second intermediate feature obtained by weighting the first intermediate feature on the y coordinate of the feature map;

[0097] The expression of the second loss function is:

[0098] L pix =|down(x inv )-x LR |

[0099] In the formula, X inv represents the super-resolution face image generated each time, and down() represents the downsampling operation on the face image;

[0100] The expression of the third loss function is:

[0101]

[0102] In the formula, represents the network intermediate layer feature map output by the very deep convolutional network VGG for large-scale image recognition,

[0103] The expression of the fourth loss function:

[0104]

[0105] Wherein, R represents the face recognition network trained with the additional angular margin loss. In this embodiment, the face recognition network uses the ResNet network trained with ArcFace. The face recognition network calculates the cosine similarity between the identity vector and the identity vector of the low-resolution image. G(z i ) represents z i The reconstructed face image is input into the progressive generative adversarial network.

[0106] Furthermore, it also includes adjusting the iteration number threshold based on the first loss function, the second loss function, the third loss function, and the fourth loss function. The preset iteration number threshold calculation formula is:

[0107] L tot =λ1L pix +λ2L div +λ3L f +λ4L id

[0108] Where λ1, λ2, λ3, and λ4 are proportional factors respectively.

[0109] Embodiment 2 is combined with Figures 2 to 4 to further illustrate the advantages of a method for reconstructing super-resolution face images from low-definition images. In this embodiment, the test data set FFHQ, which stands for Flickr-Faces-High-Quality (Flickr-Faces-HQ), was used. It was originally created as a benchmark for generative adversarial networks (GANs) and was open sourced by NVIDIA in 2019. FFHQ is a high-quality face data set, containing 70,000 high-definition face images in PNG format with a resolution of 1024×1024. It is rich and diverse in age, race, and image background, and has obvious differences. There are also many changes in facial attributes, with different ages, genders, races, skin colors, expressions, face shapes, hairstyles, facial postures, etc., covering a variety of facial peripheral accessories such as ordinary glasses, sunglasses, hats, hair accessories, and scarves. Therefore, this data set can also be used to develop some facial attribute classification or facial semantic segmentation models. FFHQ uses dlib for face alignment and cropping, and then uses an algorithm to remove some non-real faces such as statues, paintings, and photos.

[0110] 3. Experimental content

[0111] The present invention is compared with the existing human face image super-resolution method at two large super-resolution multiples of 16 times and 32 times. The super-resolution method is:

[0112] Bolei Zhou et al. published an image processing method based on multi-code generative adversarial network prior (Image Processing Using Multi-Code GAN Prior. In CVPR, 2020.), denoted as method 1;

[0113] 4. Analysis of simulation results

[0114] In the simulation experiment, the structural similarity (SSIM) and peak signal-to-noise ratio (PSNR) indicators are used to evaluate the experimental results. SSIM and PSNR are defined as:

[0115]

[0116] Where: w x , w y and σ x , σ y denote the mean and variance of x and y respectively, σ xy represents the covariance of x and y, C1 and C2 are constants. The value of structural similarity is between 0 and 1. The closer the value is to 1, the more similar the two images are.

[0117]

[0118] Where: x represents the original clear image, and z represents the restored reconstructed image. The larger the peak signal-to-noise ratio, the better the super-resolution performance.

[0119] The Frechet Inception Distance score (FID) is a measure of the distance between the feature vectors of the real image and the generated image.

[0120] FID measures the similarity of two sets of images based on the statistical similarity of the computer vision features of the original images, which are calculated using the Inception v3 image classification model. The lower the score, the more similar the two sets of images are, or the more similar their statistics are. The best case score for FID is 0.0, indicating that the two sets of images are identical. The calculation expression for FID is as follows:

[0121]

[0122] Among them, u x 、u g represents the mean of the real image and the reconstructed image, T r represents the sum of the elements on the diagonal of the matrix, ∑ x ,∑ g represents the covariance of x and g.

[0123] (1) Quantitative evaluation

[0124] In order to better evaluate the performance of these two methods, we conducted experiments on the FFHQ dataset by downsampling 16 times and 32 times into low-resolution images using Bicubic interpolation, and compared the SSIM, PSNR and FID values ​​of method 1 and the method of the present invention. The results are shown in Table 1 and Table 2 below.

[0125] Table 1:

[0126] 16 times super resolution PSNR SSIM FID Method 1 26.76 0.76 96.45 The present invention 27.44 0.78 84.10

[0127] Table 2:

[0128] 32 times super resolution PSNR SSIM FID Method 1 23.23 0.71 114.27 The present invention 23.88 0.76 98.65

[0129] Qualitative evaluation is shown in FIG2, which shows the super-resolution effect diagram between the method of the present invention and method 1; wherein,

[0130] Figure 2(a) , 2(b) Low-definition face image; Figure 3(a) is a super-resolution result of Figure 2(a) using the super-resolution method for image processing based on multi-code generative adversarial network priors disclosed in Method 1; Figure 3(b) is a super-resolution result of Figure 2(b) using the super-resolution method for image processing based on multi-code generative adversarial network priors disclosed in Method 1; Figure 4(a) is the super-resolution result of Figure 2(a) using the method of the present invention; Figure 4(b) is the super-resolution result of Figure 2(b) using the method of the present invention.

[0131] According to the above data, it can be seen that the SSIM value, PSNR value and FID value of the super-resolution face result of the method of the present invention are slightly higher than the SSIM value, PSNR value and FID value of the super-resolution results of other methods, that is, the present invention has better super-resolution effect than the prior art.

[0132] Example 3

[0133] This embodiment provides a system for reconstructing a super-resolution face image from a low-definition image, including a processor and a storage medium;

[0134] The storage medium is used to store instructions;

[0135] The processor is used to operate according to the instructions to perform the steps of the following method:

[0136] Input the low-resolution image into the pre-trained progressive generative adversarial network to obtain hidden codes and channel weights; the parameter weights of the pre-trained progressive generative adversarial network are trained by a large amount of high-definition face data;

[0137] Input the hidden code and channel weight into the iterative unit of the pre-trained progressive generative adversarial network to obtain iterative multiple hidden codes and channel weights;

[0138] Determine whether the number of iterations of the multiple hidden codes and channel weights is greater than a preset iteration number threshold, and output the multiple hidden codes and channel weights that are greater than the preset iteration number threshold;

[0139] The final super-resolution high-definition face image is generated based on the output multiple hidden codes and channel weights.

[0140] Example 4

[0141] This embodiment provides a computer-readable storage medium, on which a computer program is stored. When the program is executed by a processor, the steps of the following method are implemented:

[0142] Input the low-resolution image into the pre-trained progressive generative adversarial network to obtain hidden codes and channel weights; the parameter weights of the pre-trained progressive generative adversarial network are trained by a large amount of high-definition face data;

[0143] Input the hidden code and channel weight into the iterative unit of the pre-trained progressive generative adversarial network to obtain iterative multiple hidden codes and channel weights;

[0144] Determine whether the number of iterations of the multiple hidden codes and channel weights is greater than a preset iteration number threshold, and output the multiple hidden codes and channel weights that are greater than the preset iteration number threshold;

[0145] The final super-resolution high-definition face image is generated based on the output multiple hidden codes and channel weights.

[0146] In summary, the present invention utilizes the different knowledge representations represented by different intermediate layers in the pre-trained generative adversarial network. The features generated by the shallow network show high-level semantic features such as identity and color, and the features generated by the deep network show low-level semantic features such as structure and edge, and iteratively optimizes the multiple latent codes in groups. The multiple latent codes are divided into two groups, and are respectively placed in the sub-networks separated by different intermediate component layers in the pre-trained network for iterative optimization, so that the two parts of the multiple latent codes learn the features of different levels of semantics, and the generated super-resolution face images have a stronger sense of reality. Secondly, in the iterative optimization process, each will generate corresponding intermediate features after passing through the first sub-network of the pre-trained model, and the adaptive channel weights are used to synthesize these intermediate features into new intermediate features. Each intermediate feature is responsible for reconstructing different areas in the image. Therefore, the difference of these intermediate features is used as a constraint to make the multiple latent codes better perform their respective duties, achieve the effect of decoupling, and help to further enhance the sense of reality of the face image. Finally, to ensure that the generated super-resolution face images are not only realistic but also maintain the identity features of the original face images, the ArcFace face recognition network is introduced to calculate the feature cosine similarity between the input low-definition face and the face generated by each latent code, so as to constrain the multiple latent codes to not deviate from the identity domain of the low-definition face during iterative optimization in the latent space.

[0147] The embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of complete hardware embodiments, complete software embodiments, or embodiments in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that contain computer-usable program code.

[0148] The embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of complete hardware embodiments, complete software embodiments, or embodiments in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that contain computer-usable program code.

[0149] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0150] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0151] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0152] The above is only a preferred embodiment of the present invention. Without departing from the technical principle of the present invention, several improvements and modifications can be made, and these improvements and modifications should also be regarded as the protection scope of the present invention.

Claims

1. A method for reconstructing a super-resolution face image from a low-definition image, characterized in that: include: Input the low-resolution image into the pre-trained progressive generative adversarial network to obtain hidden codes and channel weights; the parameter weights of the pre-trained progressive generative adversarial network are trained by a large amount of high-definition face data; Input the hidden code and channel weight into the iterative unit of the pre-trained progressive generative adversarial network to obtain iterative multiple hidden codes and channel weights; Determine whether the number of iterations of the multiple hidden codes and channel weights is greater than a preset iteration number threshold, and output the multiple hidden codes and channel weights that are greater than the preset iteration number threshold; The final super-resolution high-definition face image is generated based on the output multiple hidden codes and channel weights; The method of inputting the hidden codes and channel weights into the iterative unit of the pre-trained progressive generative adversarial network to obtain iterative multiple hidden codes and channel weights includes: The hidden codes and the corresponding adaptive channel weight parameters are divided into two groups of parameters, and the two groups of parameters are respectively substituted into the pre-trained progressive generative adversarial network iteration unit to obtain iterative multiple hidden codes and channel weights. Among them, the pre-trained progressive generative adversarial network iteration unit performs Gaussian random initialization on the parameters of the hidden codes and mean initialization on the parameters of the channel weights in the first iteration; The iteration unit is composed of iterative calculation units in multiple sub-networks associated with different combination layers of the pre-trained progressive generative adversarial network, and optimizes hidden codes and adaptive channel weights through back propagation of multiple loss functions during the iteration process to obtain multiple hidden codes and channel weights that are iteratively optimized multiple times; The two groups of parameters are respectively a first group of parameters and a second group of parameters; the combination layer of the pre-trained progressive generative adversarial network includes: a first combination layer, a second combination layer and a third combination layer; The method of dividing the latent code and the corresponding adaptive channel weight parameters into two groups of parameters, and substituting the two groups of parameters into the pre-trained progressive generative adversarial network iteration unit to obtain iterative multiple latent codes and channel weights includes: The first group of parameters is iterated on the first subnetwork and the second subnetwork separated by the first combination layer, and the second group of parameters is iterated on the first subnetwork and the second subnetwork separated by the second combination layer, and two groups of latent codes and weight parameters are output; wherein each group of latent codes is input to the first subnetwork before the corresponding combination layer, each latent code generates a corresponding first intermediate feature through the first subnetwork, and the first intermediate features generated by each group of latent codes are combined into second intermediate features according to the adaptive channel weights, and then all the second intermediate features are accumulated and fused to generate third intermediate features; the third intermediate features are input to the second subnetwork to generate a face image, and the latent codes and adaptive channel weights are optimized by back-propagation through multiple loss functions, respectively, to obtain two groups of latent codes and adaptive channel weights after iterative optimization; The two sets of hidden codes and weight parameters are iterated through the first sub-network and the second sub-network of the third combination layer respectively, and multiple hidden codes and adaptive channel weights are output.

2. The method for reconstructing a super-resolution face image from a low-definition image according to claim 1, characterized in that: The pre-trained progressive generative adversarial network is preset with an iteration number threshold and a Gaussian random initialization parameter.

3. The method for reconstructing a super-resolution face image from a low-definition image according to claim 1, characterized in that: Determining whether the number of iterations of multiple hidden codes and channel weights is greater than a preset iteration number threshold also includes: For multiple hidden codes and adaptive channel weights whose iteration number is less than a preset iteration number threshold, re-input the iteration unit to update until the iteration number is greater than the preset iteration number threshold, and then output the updated multiple hidden codes and adaptive channel weights; The final super-resolution high-definition face image is obtained by super-resolution calculation based on the updated multiple hidden codes and adaptive channel weights; There are multiple preset iteration number thresholds, which respectively control the output and iteration loop of the iterative calculation units in the multiple sub-networks in rounds; multiple conditional expressions for controlling the output of the iterative calculation units in the multiple sub-networks in rounds are as follows: Wherein, l represents the current iteration round, L represents the total number of iteration rounds, k1 and k2 respectively represent the iteration rounds for iterative optimization in the combined layers of the l1-th and l2-th layers of the GAN, and l n represents the index position of the intermediate layer of the network, and the intermediate layer index l n satisfies the relationship of l1 < l2 < l3; wherein, l1 represents the index position of the first combined layer, l2 represents the index position of the second combined layer, and l3 represents the index position of the third combined layer and is processed iteratively in sequence until the preset upper limit of the number of iterations L is reached; The expressions of multiple hidden codes and adaptive channel weights after update iteration are as follows: In the formula, z (n) is the multiple hidden code, α (n) is the adaptive channel weight, N is the total number of hidden codes, L is the total loss function, X Inv is the final super-resolution high-definition face image, X LR For low-resolution images.

4. The method for reconstructing a super-resolution face image from a low-definition image according to claim 1, characterized in that: The multiple loss functions include: a first loss function, a second loss function, a third loss function, and a fourth loss function; The first loss function is a difference calculation function between the first intermediate feature and the second intermediate feature of the first sub-network; The second loss function is the mean square error loss; The third loss is the perceived loss; The fourth loss function is a cosine similarity calculation function between the identity vector of the generated reconstructed image and the identity vector of the low-resolution image; Among them, the expression of the first loss function is: Where i is the x-coordinate of the feature map, j is the y-coordinate of the feature map, Represents the second intermediate feature obtained by weighting the first intermediate feature generated by each hidden code on the x-coordinate of the feature map, The second intermediate feature obtained by weighting the first intermediate feature on the y coordinate of the feature map; The expression of the second loss function is: L pix =|down(x inv )-x LR | In the formula, X inv represents the super-resolution face image generated in each iteration, and down() represents the downsampling operation on the face image; The expression of the third loss function is: In the formula, Represents the network intermediate layer feature map output by the ultra-deep convolutional network VGG for large-scale image recognition. The expression of the fourth loss function is: Where R represents the face recognition network trained with the additional angular margin loss, which calculates the cosine similarity between the identity vector and the identity vector of the low-resolution image, G(z i ) represents z i The reconstructed face image is input into the progressive generative adversarial network.

5. The method for reconstructing a super-resolution face image from a low-definition image according to claim 4, characterized in that: It also includes adjusting the iteration number threshold based on the first loss function, the second loss function, the third loss function, and the fourth loss function. The preset iteration number threshold calculation formula is: L tot =λ1L pix +λ2L div +λ3L f +λ4L id Where λ1, λ2, λ3, and λ4 are proportional factors respectively.

6. A system for reconstructing super-resolution facial images from low-definition images, characterized in that: including processor and storage medium; The storage medium is used to store instructions; The processor is used to operate according to the instructions to execute the steps of the method according to any one of claims 1 to 5.

7. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the steps of the method according to any one of claims 1 to 5 are implemented.

Images