A realistic image reconstruction method based on latent space embedding extension
By training the intradomain encoder and embedding adversarial optimization algorithm using a two-stage method, the problem of inaccurate inverse mapping in GANs during real image reconstruction is solved, achieving high-quality image reconstruction and semantic editing capabilities.
Patent Information
- Application Number
- CN202111235541.9
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2021-10-22
- Publication Date
- 2024-11-08
- Estimated Expiration
- 2041-10-22
AI Technical Summary
Existing Generative Adversarial Networks (GANs) suffer from encoder overfitting and inaccurate inverse mapping in real image reconstruction, resulting in significant differences between the reconstructed image and the real image. Furthermore, they are not suitable for pre-trained GANs, increasing network parameters and computational costs.
A two-stage approach is adopted. First, the in-domain encoder is trained to align the latent code in the pixel domain and semantic domain. ResNet50 is used as the encoder structure, and the encoder network is optimized by combining perceptual loss and pixel-wise loss. Then, the latent code is iteratively optimized by embedding adversarial optimization algorithm to expand the latent space of StyleGAN.
It achieves accurate inverse mapping and high-quality reconstruction of real images in the latent space, ensuring that the latent code after inverse mapping can be semantically edited, thus improving the accuracy and efficiency of image reconstruction.
Smart Images

Figure CN113947550B_ABST
Abstract
Description
Technical Field
[0001] The present invention belongs to the field of computer vision, and in particular relates to a real image reconstruction method based on latent space embedding extension. Background Art
[0002] The latent space of Generative Adversarial Network (GAN) has rich semantic information. However, it is difficult to use the learned semantic information for real image editing. The purpose of real image reconstruction technology is to find a latent code in the latent space of Generative Adversarial Network (GAN) that can accurately restore the real image. More importantly, it can perform meaningful semantic manipulation by changing the latent code.
[0003] With the deepening application of generative adversarial networks (GANs) in image synthesis, it has become possible to synthesize high-quality and high-fidelity images. StyleGAN, as a style-based generator architecture, generates high-resolution images with high visual quality. Because it has a disentangled latent space w, which is obtained by mapping the initial latent space z through a multi-layer perceptron network, it has certain semantic editing capabilities. For this reason, the inverse mapping of StyleGAN can be used to obtain the latent code of the real image in the latent space, and the corresponding semantic manipulation can be performed by changing the latent code. At present, some research work has been carried out. Dumoulin V et al. proposed to train an encoder with GAN. The encoder learns to map the generated image to its original latent vector, but the encoder may be overfitted, which makes the image reconstruction significantly different from the real image. In addition, this method is not suitable for pre-trained GAN and must be trained with GAN at the same time, which increases the network parameters and computational cost. The method described by Luo et al. is to train the encoder after pre-training GAN, without training with GAN, but it cannot accurately achieve the accurate mapping of the real image in the GAN latent space, and there is a significant difference in the image quality of the inverse mapping. Summary of the invention
[0004] In summary, in response to the problems of the existing methods, the present invention proposes a real image reconstruction method based on latent space embedding expansion, which completes the inverse mapping of the real image to the latent space in two stages. In the first stage, an intra-domain encoder is trained to align the encoded latent code in the pixel domain and the semantic domain. In the second stage, the latent code obtained by the encoding network is used to obtain a preliminary restored real image. Through embedding optimization, the latent space of StyleGAN is expanded, thereby achieving accurate inverse mapping of the real image.
[0005] A method for real image reconstruction based on latent space embedding extension, comprising the following steps:
[0006] Step 1: Obtain training image dataset and preprocess, and get image labels;
[0007] Step 2: Input the training image data and labels obtained in step 1 into the established in-domain encoder network for training. The process includes:
[0008] Step 21, image inverse mapping: Use ResNet50 as the main structure of the encoder to encode the input image, and the dimension of the latent code obtained is consistent with the label dimension of the image;
[0009] Step 22, image reconstruction: send the generated latent code to the pre-trained StyleGAN2 generator to obtain the reconstructed image;
[0010] Step 23, encoder training: optimize the encoder network by combining the perceptual loss of the input image and the reconstructed image, the pixel-by-pixel loss, and the L2 norm loss between the latent codes;
[0011] Step 3: Input the real image into the trained in-domain encoder network to obtain the latent code of the real image, and use this latent code as the initial value of the embedding adversarial optimization algorithm to perform iterative optimization, and finally obtain the reconstruction of the input image; the embedding adversarial optimization process is as follows:
[0012] Starting from the latent code w generated by the pre-trained ResNet50 in-domain encoder, search for the optimized vector w * , by minimizing the difference between the input real image and w * The perceptual loss and pixel-wise loss between the generated reconstructed images are used to iteratively optimize the latent code.
[0013] Furthermore, the training image dataset of step 1 is generated by the generative model of StyleGAN2.
[0014] Furthermore, the step 1 also includes using a Yolov3 network with a confidence level of 0.98 to screen the generated training image data set and remove images with low generated quality.
[0015] Furthermore, the label of the image in step 1 is obtained by passing the normal random sampling value through the mapping network of StyleGAN2, and the dimension is 18*512.
[0016] Furthermore, the step 2 combines the L2 norm loss between latent codes, the perceptual loss of the input image and the generated image, and the pixel-wise loss.
[0017] Furthermore, in step 3, the latent code generated by the encoder model trained in step 2 is used as the initial value for latent code optimization.
[0018] Furthermore, the step 3 combines the perceptual loss and the pixel-wise loss of the input and output images.
[0019] The present invention adopts the above technical solution to produce the following technical effects:
[0020] The real image reconstruction method based on latent space embedding extension provided by the present invention can accurately find the accurate inverse mapping of the input real image in the extended latent space through the intra-domain coding network plus the embedding adversarial optimization algorithm, and obtain an image with good reconstruction effect. While highly restoring the real image, it also ensures that the latent code after the inverse mapping can be semantically edited by changing the latent code. BRIEF DESCRIPTION OF THE DRAWINGS
[0021] Figure 1 Schematic diagram of the principle of the real image reconstruction method based on latent space embedding extension of the present invention. DETAILED DESCRIPTION
[0022] The technical solution of the present invention is further described in detail below in conjunction with the accompanying drawings.
[0023] The overall structure of the present invention is as follows Figure 1 As shown in the figure, a real image reconstruction method based on latent space embedding extension is proposed. This method specifically includes the following steps:
[0024] Step 1: Obtain training image data and preprocess.
[0025] Since the network needs to be trained multiple times and adapt to various application scenarios, the amount of training data prepared needs to meet certain requirements. The latent vector is obtained through normal random sampling. Through the StyleGAN2 generator, 110k images are obtained. Since some generated images are of low quality, the generated 110k images are screened through the Yolov3 network. The network confidence of Yolov3 is 0.98. Finally, 100k images are screened. The image label is the value of normal random sampling obtained through the mapping network of StyleGAN2, with a dimension of 18*512.
[0026] Step 2. First, the encoder is designed with a ResNet50 network. The input of the network is a 256*256 generated image. The goal is to train an encoder to map the real image to the k*512-dimensional w space of StyleGAN2, where k is the number of style inputs of the generator. For example, a generator that can synthesize images at a resolution of 1024×1024 corresponds to 18 w spaces corresponding to the same style inputs. Therefore, the last three fully connected layers of the ResNet50 network are removed, and three convolutional layers are added. Each convolutional layer contains 144 convolutional kernels of size 3*3. The output of the network becomes 18*512 dimensions, which is convenient for the alignment of the latent code. The specific network structure is as follows: Figure 1 shown.
[0027] Secondly, the in-domain encoder network established by inputting the acquired preprocessed training image data and labels is trained. The process is as follows:
[0028] Step 21. During the encoder training process, the generator of StyleGAN2 is fixed and only the encoder network is optimized. In order to measure the similarity between the input image and the embedded image during the optimization process, a loss function is used in combination. This function is the perceptual loss obtained by VGG16 and pixel-level MSE loss The perceptual loss and pixel-level MSE loss are chosen because the pixel-level MSE loss cannot find high-quality inverse mapping latent codes, while the perceptual loss acts like a regulator that guides the optimization into the correct area of the latent space.
[0029] L part =min w λ percept L percrpt (G(w),I1)+λ mse L mse (G(w),I1)
[0030] where I1∈R n×n×3 is the input image, G is the pre-trained StyleGAN2 generator, N is the number of scalars in the image (N = n × n × 3), w is the latent code to be optimized, and λ percept ,λ mse is the weight of perceptual loss and pixel-level MSE loss. percept ,λ mse is 1, for perceptual loss
[0031]
[0032] Where I1 is the input image, I2 is the reconstructed image generated by the encoded latent code w through StyleGAN2, I1, I2∈R n ×n×3 , F j are the conv1_1, conv1_2, conv3_2, and conv4_2 layer outputs of VGG16, respectively. j is the scalar number of the j-th layer output, and all λ obtained empirically j =1 has good performance.
[0033] For MSE loss, it is as follows
[0034]
[0035] where I1∈R n×n×3 is the input image, G is the pre-trained StyleGAN2 generator, N is the number of scalars in the image (N = n × n × 3), and w is the latent code to be optimized.
[0036] As can be seen from step 1, the dataset used in the experiment is an image generated by the StyleGAN2 pre-training model. In this way, when training the encoder, the latent code of the input real image can be obtained for supervised training. In this way, for each image in the dataset, its latent code representation is obtained. In this way, the L2 norm loss of the real latent code and the encoded latent code can be expressed as
[0037]
[0038] The total loss function is
[0039]
[0040] The input of the network is the image I1 generated by the StyleGAN2 pre-trained network. The generated image is sent to the ResNet50 encoding network to obtain the latent code w with a dimension of 18*512. The latent code w is then sent to the generator G of StyleGAN2 to obtain the reconstructed image I2, forming a training pair (I1, I2). Since the input image I1 is obtained through the pre-trained StyleGAN2 model, the label of the image I1, that is, the latent code of the image I1, can be obtained. The L2 norm loss of the latent code, plus the perceptual loss and pixel-by-pixel MSE loss calculated for the previous training pair (I1, I2), are used to jointly optimize the encoder to obtain the distribution of the latent code w with the best reconstruction effect.
[0041] Step 3: The embedding optimization process of the present invention is to embed the given image into the manifold after the StyleGAN2 generator, starting from the latent code w generated by the pre-trained ResNet50 in-domain encoder, and search for the optimized vector w * , so that the given image is consistent with the image from w * The loss function of the similarity between the generated images is minimized. The strategy of the present invention tends to produce reverse latent codes with higher reconstruction quality and converges within a specified time. Further, in a fixed number of iterations, the latent codes after the ResNet50 encoder are further optimized by gradient descent, and the joint perceptual loss and pixel-by-pixel MSE loss are used as the optimization function.
[0042]
[0043] where w * is the optimal latent code, λ percept is a scalar weighting the VGG16 perceptual loss, G is the pre-trained StyleGAN2 generator, and w inv is the latent code to be optimized, which is obtained by pre-training ResNet50 encoder, I∈R n×n×3 is the input image, λ mse is a scalar that assigns weights to the pixel-level mean square error loss. In the experiment, λ percept , mse The scalar value of is 1.
[0044] In summary, the present invention proposes a real image reconstruction method based on latent space embedding expansion, which highly restores the real image while ensuring that the latent code after inverse mapping can be semantically edited by changing the latent code. First, an intra-domain encoder is trained. The encoder not only emphasizes the alignment of the latent code, but also introduces perceptual loss and pixel-by-pixel loss to align the encoded latent code in the pixel domain and the semantic domain. In order to further improve the restoration quality of the reconstructed image, an embedding optimization algorithm is proposed to realize the expansion of the latent space. The latent code obtained by the intra-domain encoder is optimized as the initial optimization value to fine-tune the latent code generated by the encoder and better restore the target image.
[0045] The above description is only a preferred embodiment of the present invention, and the protection scope of the present invention is not limited to the above embodiment. Any equivalent modifications or changes made by ordinary technicians in this field based on the contents disclosed by the present invention should be included in the protection scope recorded in the claims.
Claims
1. A method for real image reconstruction based on latent space embedding extension, characterized by: The method comprises the following steps: Step 1: Obtain training image dataset and preprocess, and get image labels; Step 2: Input the training image data and labels obtained in step 1 into the established in-domain encoder network for training. The process includes: Step 21, image inverse mapping: Use ResNet50 as the main structure of the encoder to encode the input image, and the dimension of the latent code obtained is consistent with the label dimension of the image; Step 22, image reconstruction: send the generated latent code to the pre-trained StyleGAN2 generator to obtain the reconstructed image; Step 23, encoder training: optimize the encoder network by combining the perceptual loss of the input image and the reconstructed image, the pixel-by-pixel loss, and the L2 norm loss between the latent codes; For the perceived loss Where I1 is the input image, I2 is the reconstructed image generated by the encoded latent code w through StyleGAN2, I1, I2∈R n×n×3 , F j are the conv1_1, conv1_2, conv3_2, and conv4_2 layer outputs of VGG16, respectively. j is the scalar number of outputs of the jth layer, λ j =1; For MSE loss, it is as follows: where I1∈R n×n×3 is the input image, G is the pre-trained StyleGAN2 generator, N is the number of scalars in the image (N = n × n × 3), and w is the latent code to be optimized; The L2 norm loss of the real latent code and the encoded latent code is expressed as The total loss function is Step 3: Input the real image into the trained in-domain encoder network to obtain the latent code of the real image, and use this latent code as the initial value of the embedding adversarial optimization algorithm to perform iterative optimization, and finally obtain the reconstruction of the input image; the embedding adversarial optimization process is as follows: Starting from the latent code w generated by the pre-trained ResNet50 in-domain encoder, search for the optimized vector w * , by minimizing the difference between the input real image and w * The perceptual loss and pixel-wise loss between the generated reconstructed images are used to iteratively optimize the latent code.
2. The method for real image reconstruction based on latent space embedding extension according to claim 1, characterized in that: The training image dataset of step 1 is generated by the generative model of StyleGAN2.
3. The method for real image reconstruction based on latent space embedding extension according to claim 2, characterized in that: The step 1 also includes using a Yolov3 network with a confidence level of 0.98 to screen the generated training image data set and remove images with low generated quality.
4. The method for real image reconstruction based on latent space embedding extension according to claim 1, characterized in that: The label of the image in step 1 is obtained by normal random sampling through the mapping network of StyleGAN2, and the dimension is 18*512.
5. The method for real image reconstruction based on latent space embedding extension according to claim 1, characterized in that: The step 2 combines the L2 norm loss between latent codes, the perceptual loss of input and generated images, and the pixel-wise loss.
6. The method for real image reconstruction based on latent space embedding extension according to claim 1, characterized in that: In step 3, the latent code generated by the encoder model trained in step 2 is used as the initial value for latent code optimization.
7. The method for real image reconstruction based on latent space embedding extension according to claim 1, characterized in that: The step 3 combines the perceptual loss and the pixel-wise loss of the input and output images.
