A real image aesthetic enhancement method based on mGANprior

Abstract

The present invention discloses a real image aesthetic enhancement method based on mGANprior. For a real image to be enhanced in aesthetics, a corresponding type of PGGAN pre-trained generation model is selected, and the type of aesthetic effect to be enhanced is determined; a cascade segmentation module method is used to perform semantic segmentation on the real image; an mGANprior method is used to obtain an inversely mapped image; and according to the aesthetic style to be enhanced, corresponding degradation transformations are performed on the real image and the inversely mapped image, and the loss is calculated, and the final latent vector and the image after the image aesthetic style is enhanced are obtained through gradient descent optimization. enh ; The method of the present invention realizes the aesthetic enhancement of real images, which not only retains the original information of the image to the greatest extent, but also modifies the image in a controllable aesthetic style. The present invention proposes a loss function of degenerate transformation based on aesthetic factors, which can generate an aesthetic blur effect on the real image.

Description

Technical Field

[0001] The present invention relates to the field of inverse mapping of a generative adversarial network, semantic segmentation of an image, and image aesthetic enhancement. Specifically, a real image is inversely mapped to a latent space of a generative adversarial network, and an aesthetic degradation transformation is constructed for two aesthetic styles for loss calculation, thereby achieving the effect of enhancing the beauty of the real image. Background Art

[0002] Research based on generative adversarial networks (GANs) has made great breakthroughs and progress in recent years. Scholars have developed a large number of high-quality derivative models based on GANs, such as PGGAN, StyleGAN, and BigGAN. PGGAN is the first generative model that proposes a progressive training method to obtain high-quality generated images. This model can eventually obtain 1024×1024 high-quality images through progressive training. PGGAN provides pre-trained generative models in multiple scenarios, such as churches, towers, bridges, bedrooms, etc.

[0003] The GAN model maps the latent space to the image space, while the GAN inverse mapping is the inverse process, that is, establishing a mapping from the image space to the latent space. The GAN inverse mapping aims to map the real image back to the latent space of the pre-trained GAN model, and then the generative model can reconstruct the image from the latent space obtained by the inverse mapping. mGANprior is a GAN inverse mapping method that can reconstruct real images with high quality. It has been proven that this method has excellent inverse mapping effects on generative models of various indoor and outdoor scenes. At the same time, before calculating the loss between the generated image and the real image, mGANprior implements image processing such as coloring, super-resolution reconstruction, and image restoration of the real image by applying different degradation transformations to the image.

[0004] Semantic segmentation is a deep learning algorithm that associates a label or category with each pixel of an image. It is used to identify sets of pixels that make up distinguishable categories. For example, a bridge landscape photo needs to identify bridges, streams, grass, trees, and the sky. The Cascade Segmentation Module method is a general semantic segmentation method proposed in the paper "Scene Parsing through ADE20K Dataset". It parses the scene into materials, objects, and object parts in a cascade manner, and can be applied to semantic segmentation in different scenarios.

[0005] Based on the above background, if a degradation transformation method is designed for 14 types of aesthetic styles, the aesthetic enhancement of real images based on GAN inverse mapping can be achieved. The prerequisite is that the degradation transformation is known and differentiable. For example, the degradation transformations corresponding to image colorization, super-resolution reconstruction, and image restoration are grayscale, downsampling, and image cropping, which are all known and differentiable. However, there is still no differentiable transformation method for aesthetic factors based on current research results. Therefore, the present invention intends to design a degradation transformation method for aesthetic styles, thereby achieving the aesthetic enhancement of real images based on GAN. Summary of the invention

[0006] In view of the above problems, the present invention proposes a real image aesthetic enhancement method based on mGANprior, designs different degradation transformation methods for two fuzzy aesthetic styles, namely shallow depth of field and motion blur, to achieve fuzzy aesthetic enhancement of real images. The technical solution of the present invention includes the following steps:

[0007] Step 1: Select a real image I to be enhanced, select the corresponding type of PGGAN pre-trained generative model, and determine whether the image needs to be enhanced with shallow depth of field or motion blur.

[0008] Step 2: Use the Cascade Segmentation Module method to perform semantic segmentation on the real image I and extract the main pixels. And according to the required enhanced aesthetic effect (shallow depth of field or motion blur), different binary matrices m are obtained.

[0009] Step 3: Use the mGANprior method to obtain the inverse mapped image I inv .

[0010] Step 4: According to the aesthetic style to be enhanced, compare the real image I and the inverse mapped image I inv Make the corresponding degradation transformation and calculate the loss, then optimize the latent vector Z according to the gradient descent i,( i ∈(1，n)) , the optimized Z i,(i∈(1，n)) As input again, the new inverse mapped image I is obtained through the mGANprior method in step 3 inv , and calculate the loss until the loss has no downward trend for ten consecutive iterations, then stop training and get the final latent vector Z i,(i∈(1，n)) Image I after image aesthetic style enhancement enh The loss function combines the mean square error loss (MSE) with the perceptual feature reconstruction loss.

[0011] Furthermore, the specific method of step 3 is as follows:

[0012] The mGANprior method is used to split the generative network of the generative model selected in step 1 into two layers from the specified layer. The layer and the previous network layer are the pre-network G1, and all networks after this layer are the post-network G2. The specified layer level is selected according to the needs, and the effect of inverse mapping is positively correlated with the layer depth. Randomly generate n hidden vectors Z i,(i∈(1，n)) As the input of the front network G1, n feature maps are obtained through the front network G1, and the obtained feature maps are combined based on the adaptive channel importance and input into the back network G2 to obtain the generated image I inv .

[0013] Furthermore, the specific method of step 4 is as follows:

[0014] (1) For the real image I and the inverse mapped image I inv Apply image degradation transformation to obtain X and X inv .

[0015] If the shallow depth of field is used for aesthetic enhancement, the following shallow depth of field degradation transformation is performed. According to m, the background part of the real image I and the inverse mapped image I inv The main part of the pixels is downsampled. The degradation transformation formula is as follows:

[0016] X=I*m+down(I*(1-m))(1)

[0017] X inv =down(I inv *m)+I inv *(1-m) (2)

[0018] If the motion blur aesthetic is enhanced, the following motion blur degradation transformation is performed. Downsample the pixels of the main part of the real image I according to m. The degradation transformation formula is as follows:

[0019] X=down(I*m)+I*(1-m)(3)

[0020] X inv =I inv (4)

[0021] (2) Calculate X and X inv MSE and perceptual feature loss:

[0022]

[0023] in For X and X inv MSE loss, where φ(·) is the perceptual feature extractor, φ(X) and φ(X inv ) are X and X respectively inv Perceptual features, ||φ(X), φ(Xinv )||1 is φ(X) and φ(X inv )’s L1 distance.

[0024] (3) Optimize Z using gradient descent i,(i∈(1，n)) .

[0025] (4) Iterative training. Using the mGANprior method in step 3, the optimized Z i,(i∈(1，n)) It is used as the input of the front network G1 again to obtain n feature maps; according to the principle of adaptive channel importance, the n feature maps are combined and input into G2 to obtain a new inverse mapped image I inv ;

[0026] (5) Repeat the above steps (1)-(4) to perform the degradation transformation and calculate the loss. According to the gradient descent, Z i,(i∈(1，n)) Optimize and iterate the training. Stop the training until the loss does not decrease for ten consecutive iterations, and get the I with enhanced fuzzy beauty. enh Using the shallow depth of field degenerate transform will result in an image with enhanced shallow depth of field aesthetics, and using the motion blur degenerate transform will result in an image with enhanced motion blur aesthetics.

[0027] The beneficial effects of the present invention are as follows:

[0028] 1. Using the GAN inverse mapping method mGANprior, the aesthetic enhancement of real images is achieved. It not only retains the original information of the image to the greatest extent, but also modifies the image in a controllable aesthetic style.

[0029] 2. The blur effects in aesthetic styles, such as shallow depth of field and motion blur, are the two most difficult effects to achieve with GAN, because the GAN model will try to eliminate blur during the training process. This patent proposes a loss function of degenerate transformation based on aesthetic factors, which can generate aesthetic blur effects on real images. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] Figure 1 It is a flow chart for implementing the method of the present invention. DETAILED DESCRIPTION

[0031] The specific embodiments of the present invention are further described in detail below in conjunction with the accompanying drawings.

[0032] like Figure 1As shown, the present invention proposes a real image aesthetic enhancement method based on mGANprior, inputs a real image and selects a suitable PGGAN generation model, performs semantic segmentation on the image to extract the main part to be enhanced, and generates a binary matrix m according to the image and the extracted semantics, in which the main part value is 1 and the background part value is 0. Use mGANprior to obtain the inverse mapped image, perform degradation transformation on the real image and the inverse mapped image and calculate the loss, perform parameter optimization and iterative training according to gradient descent, and finally obtain an aesthetically enhanced image. The detailed steps are as follows.

[0033] Step 1: Select a real image I to be enhanced, and manually determine whether the image belongs to a category in the PGGAN pre-trained generative model. For example, for an image containing a bridge, select the PGGAN corresponding bridge generative model. If it does not belong to any category in the PGGAN pre-trained generative model, you can select any PGGAN generative model. And determine whether you want to enhance the image with shallow depth of field or motion blur.

[0034] Step 2: Use the cascade segmentation module to perform semantic segmentation on the image and extract the main pixels. From this, a binary matrix m is obtained. If the shallow depth of field aesthetics is enhanced, the value of the part that needs to be clearly imaged in m is set to 1, and the rest of the values ​​are set to 0; if the motion blur aesthetics is enhanced, the value of the part that needs to be blurred in m is set to 1, and the rest of the values ​​are set to 0.

[0035] Depth of field refers to the range of clear images before and after the focus point. All elements in this range can be clearly imaged, while elements outside the range will gradually become blurred. Shallow depth of field means that only part of the image is in focus. Motion blur is a static scene or a series of pictures, such as fast-moving objects in movies or animations, causing obvious blurry dragging marks.

[0036] Step 3: Obtain the inverse mapping image I through the mGANprior model inv .

[0037] (1) The PGGAN generation network uses the 8th layer as the dividing line, with the first to eighth layers as the pre-network G1 and the subsequent networks as the post-network G2;

[0038] (2) Generate 30 latent vectors Z i,(i∈(1,30)) ;

[0039] (3) Z i,(i∈(1,30)) Input G1 to get 30 feature maps;

[0040] (4) According to the principle of adaptive channel importance, the 30 feature maps are combined and input into G2 to obtain the inverse mapped image I inv .

[0041] Step 4: Compare the real image I and the inverse mapped image I according to the aesthetics to be enhanced inv Apply different degenerate transformations to obtain X and X inv , calculate X and X inv The loss is used as the training loss, and Z is optimized by gradient descent i,(i∈(1,30)) .

[0042] (1) For the real image I and the inverse mapped image I inv Apply image degradation transformation to obtain X and X inv .

[0043] If the shallow depth of field is used for aesthetic enhancement, the following shallow depth of field degradation transformation is performed. According to m, the background part of the real image I and the inverse mapped image I inv The main part of the pixels is downsampled. The degradation transformation formula is as follows:

[0044] X=I*m+down(I*(1-m))(1)

[0045] X inv =down(I inv *m)+I inv *(1-m) (2)

[0046] If the motion blur aesthetic is enhanced, the following motion blur degradation transformation is performed. Downsample the pixels of the main part of the real image I according to m. The degradation transformation formula is as follows:

[0047] X=down(I*m)+I*(1-m)(3)

[0048] X inv =I inv (4)

[0049] (2) Calculate X and X inv MSE and perceptual feature loss

[0050]

[0051] in For X and X inv MSE loss, where φ(·) is the perceptual feature extractor, φ(X) and φ(X inv ) are X and X respectively inv Perceptual features, ||φ(X), φ(X inv )||1 is φ(X) and φ(X inv )’s L1 distance.

[0052] (3) Optimize Z using gradient descent i,(i∈(1，30)) .

[0053] (4) Iterative training. Using the mGANprior method in step 3, the optimized Z i,(i∈(1，30)) It is used as the input of the front network G1 again to obtain 30 feature maps;

[0054] According to the principle of adaptive channel importance, the 30 feature maps are combined and input into G2 to obtain the new inverse mapped image I inv ,

[0055] (5) Repeat the above steps (1)-(4) to perform the degradation transformation and calculate the loss. i，(i∈(1，30)) Optimize and iterate the training. Stop the training until the loss does not decrease for ten consecutive iterations, and get the I with enhanced fuzzy beauty. enh Using the shallow depth of field degenerate transform will result in an image with enhanced shallow depth of field aesthetics, and using the motion blur degenerate transform will result in an image with enhanced motion blur aesthetics.

Claims

1. A real image aesthetic enhancement method based on mGANprior, characterized in that: The steps include: Step 1: Select a real image I to be enhanced, select the corresponding type of PGGAN pre-trained generative model, and determine whether the image needs to be enhanced with shallow depth of field or motion blur; Step 2: Use the cascade segmentation module method to perform semantic segmentation on the real image I and extract the main pixels; And according to the aesthetic effect to be enhanced, different binary matrices m are obtained; Step 3: Use the mGANprior method to obtain the inverse mapped image I inv ; Step 4: According to the aesthetic style to be enhanced, compare the real image I and the inverse mapped image I inv Make the corresponding degradation transformation and calculate the loss, then optimize the latent vector z according to the gradient descent i,i∈(1,n) , the optimized z i,i∈(1,n) As input again, the new inverse mapped image I is obtained through the mGANprior method in step 3 inv , and calculate the loss until the loss has no downward trend for ten consecutive iterations, then stop training and get the final latent vector z i,i∈(1,n) Image I after image aesthetic style enhancement enh ; The loss function combines the mean square error loss MSE with the perceptual feature reconstruction loss; Step 3: The mGANprior method is used to split the generative network of the generative model selected in step 1 into two layers from the specified layer. The layer and the previous network layer are the pre-network G1, and all networks after the layer are the post-network G2. The specified layer level is selected according to the needs, and the effect of inverse mapping is positively correlated with the layer depth; randomly generate a number n of latent vectors z i,i∈(1,n) As the input of the front network G1, n feature maps are obtained through the front network G1, and the obtained feature maps are input into the back network G2 based on the adaptive channel importance combination to obtain the generated image I inv ; Step 4: (1) For the real image I and the inverse mapped image I inv Apply image degradation transformation to obtain X and X inv ; If the shallow depth of field is used for aesthetic enhancement, the following shallow depth of field degradation transformation is performed; according to m, the background part of the real image I and the inverse mapped image I inv The pixels of the main part are downsampled; the degradation transformation formula is as follows: X=I*m+down(I*(1-m)) (1) X inv =down(I inv *m)+I inv *(1-m) (2) If the motion blur aesthetics are enhanced, the following motion blur degradation transformation is performed; the pixels of the main part of the real image I are downsampled according to m; the degradation transformation formula is as follows: X=down(I*m)+I*(1-m) (3) X inv =I inv (4) (2) Calculate X and X inv MSE and perceptual feature loss: in For X and X inv MSE loss, where φ(·) is the perceptual feature extractor, φ(X) and φ(X inv ) are X and X respectively inv Perceptual features, ||φ(X),φ(X inv )||1 is φ(X) and φ(X inv )’s L1 distance; (3) Optimize z using gradient descent i,i∈(1,n) ; (4) Iterative training; Using the mGANprior method in step 3, the optimized z i,i∈(1,n) It is used as the input of the front network G1 again to obtain n feature maps; according to the principle of adaptive channel importance, the n feature maps are combined and input into G2 to obtain a new inverse mapped image I inv ; (5) Repeat the above steps (1)-(4) to perform the degradation transformation and calculate the loss. i,i∈(1,n) Optimize and iterate training; stop training until the loss does not show a downward trend for ten consecutive iterations, and get I with enhanced fuzzy beauty enh ; Using the shallow depth of field degenerate transform will obtain an image with enhanced shallow depth of field beauty, and using the motion blur degenerate transform will obtain an image with enhanced motion blur beauty.

Images