Data enhanced CyclGAN regularization image style migration model construction method

Abstract

The invention discloses a data enhancement CycleGAN regularization image style migration model construction method so as to realize migration from a non-paired natural landscape photo to a traditional ink and wash painting style. According to the method, a double-generator-discriminator structure is designed, so that the mapping constraint of the one-way GAN model is effectively improved; in order to better learn and migrate the style of the traditional wash painting, new regularization loss is designed, including total variation loss and a cyclic consistency loss function improved by combining multi-scale structural similarity, so that the accuracy and quality of image migration are improved, and the characteristics and artistic conception of the traditional wash painting are better captured; aiming at the lack of an ink and wash painting pairing data set, a data enhancement technology is adopted to increase the quantity and diversity of real data and generated data, so that the performance of a generator and the accuracy of a model are improved, and the peak signal-to-noise ratio (PSNR) and the structural similarity (SSIM) are both higher than those in the prior art.

Description

Technical Field

[0001] The method of the invention relates to a method for constructing an image style transfer model, in particular to a method for constructing an image style transfer model using data-enhanced CycleGAN regularization. Background Art

[0002] In recent years, image style transfer technology based on deep learning has been widely used in computer vision, image processing, and artistic creation. Its background can be traced back to the style transfer technology in the field of art, that is, applying a painting style to another image. At present, there are two main mainstream models, namely style transfer based on convolutional neural networks and style transfer based on generative adversarial networks. The former uses convolutional neural networks as feature extractors, which can effectively retain the content information of the image and better integrate the texture and color features of the target style. However, the backpropagation optimization process is slow, the model training time is long, and it is often difficult to control the details of the generated image, such as some subtle texture and color changes. The generative adversarial network adopts the idea of ​​adversarial game, and controls the details and style features of the generated image by controlling the parameters of the generator and the discriminator. It can generate more realistic, natural, and detailed images, and has a good style transfer effect, but the instability of its model training affects the visual effect and quality of the generated image.

[0003] The style of traditional ink painting (Cursive calligraphy of the Tang Dynasty, flower and bird painting of the Song Dynasty, landscape painting of the Yuan Dynasty, etc.) not only has high artistic value, but also has a profound impact on the painting and artistic creation of later generations. Therefore, applying the artistic style of traditional ink painting to digital image processing based on image style transfer technology to create images with more artistic sense and visual effects has extremely high application value and research significance. However, for the traditional Chinese ink painting with unique style and elegant artistic conception, the above two style transfer methods both face the problem of lack of real matching data, which limits the authenticity and details of the generated images. Summary of the invention

[0004] The present invention aims to solve the above-mentioned technical problems existing in the prior art and provide a method for constructing an image style transfer model using data-enhanced CycleGAN regularization.

[0005] The technical solution of the present invention is: a method for constructing a data-enhanced CycleGAN regularized image style transfer model, which is carried out in the following steps:

[0006] Step 1. Collect training set images

[0007] The traditional Chinese ink landscape images and natural scenery images in the dataset are used as training sets, denoted as trainA and trainB respectively; the traditional Chinese ink landscape images are defined as A domain images, and the natural scenery images are defined as B domain images;

[0008] Step 2. Define the overall network structure

[0009] The overall network structure consists of two pairs of generator-discriminator structures, namely G A2B -D B , G B2A -D A , generator G A2B Realize the conversion of domain A images to domain B images, discriminator D B Determine whether the input image is a real B-domain image; Generator G B2A Realize the conversion of B domain to A domain image, discriminator D A Determine whether the input image is a real A domain image;

[0010] The generator G A2B , G B2A The network structure consists of three parts: encoder feature extraction, feature conversion and decoder reconstruction features. The encoder extracts features and is composed of a padding layer, a 7×7 convolutional layer Conv1-1, and two downsampling modules. The padding layer performs a reflection padding operation to fill the image with 3 rows on the top, bottom, left and right sides. The 7×7 convolutional layer Conv1-1 is a convolutional layer with 64 7×7 filters, a normalization layer BatchNorm and a Rule activation function. The downsampling module is composed of a convolutional layer with 64 3×3 filters, a BatchNorm normalization layer and Relu activation function; the feature conversion part is composed of 9 residual blocks, each of which is composed of two convolutional layers, two normalization layers, a Relu activation function and a jump connection; the decoder reconstruction feature part is composed of two upsampling modules, a 7×7 convolutional layer Conv1-2 and a padding layer, the upsampling module is a deconvolution operation Deconv, the 7×7 convolutional layer Conv1-2 is a combination of a convolutional layer with 64 7×7 filters and a tanh activation function, and the padding layer is a reflection padding operation to fill the image with 3 rows up, down, left and right;

[0011] The discriminator D A , D BUsing the PatchGAN network structure, the network first passes through a convolutional layer consisting of 64 4×4 filters and a convolutional layer Conv2-1 consisting of a LeakyRelu activation function, then passes through three feature extraction layers Conv2-2 consisting of a convolutional layer, a normalization layer, and a LeakyRelu activation function, and finally passes through a convolutional layer with a channel of 1 to obtain the predicted value;

[0012] Step 3. Design regularization loss, train the network, and obtain a network model that implements the style transfer of traditional Chinese ink painting. The specific steps are as follows:

[0013] Step 3.1 Randomly initialize the generator and discriminator network parameters;

[0014] Step 3.2 starts iteratively training the generator and discriminator, and updates the generator and discriminator network parameters; the agreed variable epoch records the number of current network training iterations, epoch is initially 0, and each round of iteration is divided into the following two parts:

[0015] Step 3.2.1: Fix the generator network parameters and adjust the discriminator D A , D B Network parameters; for the discriminator D A , the real A domain image real_a in the training set trainA is processed by data enhancement using the cutout strategy and recorded as T. The processed image is recorded as real_a_aug and then input into the discriminator D A The output of the network is recorded as pred_real_a_aug; the generator G B2A Generate fake image fake_a and use the cutout strategy for data enhancement. The processed image is recorded as fake_a_aug and input into the discriminator D A The output of the network is recorded as pred_fake_a_aug. The mean square loss function is used to calculate the loss between pred_real_a_aug and the label True, recorded as loss_DA_real, and the loss between pred_fake_a_aug and the label False, recorded as loss_DA_fake. The mean is calculated by combining loss_DA_real and loss_DA_fake to obtain the discriminator D. A Loss loss_D_A; Similarly, for the discriminator D B , the real B domain image real_b in the training set trainB is processed by the cutout strategy for data enhancement. The processed image is recorded as real_b_aug and then input into the discriminator D B The network output is recorded as pred_real_b_aug; the generator G A2BThe fake image fake_b is generated by using the cutout strategy for data enhancement. The processed image is recorded as fake_b_aug and then input into the discriminator D B The output of the network is recorded as pred_fake_b_aug. The mean square loss function is used to calculate the loss between pred_real_b_aug and the label True, recorded as loss_DB_real, and the loss between pred_fake_b_aug and the label False, recorded as loss_DB_fake. The mean is calculated by combining loss_DB_real and loss_DB_fake to obtain the discriminator D. B Loss loss_D_B; calculate the discriminator D in this round of iteration A Loss loss_D_A and discriminator D B Loss loss_D_B gradient, back propagation, and re-update the discriminator D through the Adam optimization algorithm A , D B parameter;

[0016] Step 3.2.2 Fix the discriminator network parameters and adjust the generator G A2B , G B2A Network parameters; for generator G A2B , put the real A domain image real_A into G A2B The network generates a fake image fake_B in domain B, and then inputs fake_B into the generator G. B2A , get the reconstructed A domain image rec_A, define loss_G_A2B variable record generator G A2B Total loss, generator G A2B The total loss consists of four parts: The first part calculates the generator G A2B Ordinary GAN loss, the fake image fake_B is first enhanced to obtain fake_B_aug, and then sent to the discriminator D BThe output is obtained, and the mean square loss function is used to calculate the loss between the output and the True label, which is recorded as GANloss_A; the second part uses L1loss to calculate the difference between the reconstructed A-domain image rec_A and the real A-domain image real_A, that is, the cycle consistency loss function, recorded as Cycleloss_A; the third part uses multi-scale structural similarity to measure the similarity between the reconstructed A-domain image rec_A and the real A-domain image real_A in terms of color and hue, etc., recorded as MSSSIM_A, and calculates the 1-MSSSIM_A value as the multi-scale structural similarity loss function MS_SSIMloss_A; the fourth part calculates the sum of the differences between all pixels of the generated fake image fake_B and the adjacent pixels in the vertical and horizontal directions, that is, the total variational loss TVloss_A; the above four loss functions are weighted and accumulated as the generator G A2B Total loss loss_G_A2B; Similarly, for generating G B2A , the total loss loss_G_B2A is calculated and accumulated by the above four parts; finally, the gradients of loss_G_A2B and loss_G_B2A are calculated and back-propagated, and the generator G is updated again through the Adam optimization algorithm A2B , G B2A Network parameters;

[0017] Step 3.2.3: This round of iteration ends, set epoch = epoch + 1; when the epoch reaches the set number of times, the network stops training and saves the training model model.pt file; otherwise, enter step 3.2 again and repeat the training of the discriminator and generator.

[0018] The present invention designs a dual generator-discriminator structure to effectively improve the mapping constraints of the unidirectional GAN ​​model; in order to better learn and transfer the traditional ink painting style, a new regularization loss is designed, including total variation loss and improved cycle consistency loss function combined with multi-scale structural similarity, thereby improving the accuracy and quality of image migration and better capturing the characteristics and artistic conception of traditional ink painting; in view of the lack of paired ink painting data sets, data enhancement technology is used to increase the number and diversity of real data and generated data, thereby improving the performance of the generator and the accuracy of the model, and its peak signal-to-noise ratio (PSNR) and structural similarity (SSIM) are higher than the prior art. The present invention can effectively convert natural landscape photos into images in the style of traditional ink painting, and the resulting image effect is more in line with the characteristics of real ink painting style. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] Figure 1 This is a diagram of the network structure of the generator according to an embodiment of the present invention.

[0020] Figure 22 is a diagram showing the structure of a discriminator network according to an embodiment of the present invention.

[0021] Figure 3 1 is a schematic diagram of the process and loss function of training the discriminator in an embodiment of the present invention.

[0022] Figure 4 It is a schematic diagram of the process and loss function of training the generator in an embodiment of the present invention. DETAILED DESCRIPTION

[0023] A method for constructing a data-enhanced CycleGAN regularized image style transfer model of the present invention is carried out according to the following steps:

[0024] Step 1. Collect training set images

[0025] The 1992 traditional Chinese ink landscape paintings and 6440 natural scenery images in the Chinese_Landscape_Painting2photo dataset are used as training sets, denoted as trainA and trainB respectively. The traditional Chinese ink landscape paintings are defined as A domain images, and the natural scenery images are defined as B domain images.

[0026] Step 2. Define the overall network structure

[0027] The overall network structure consists of two pairs of generator-discriminator structures, namely G A2B -D B , G B2A -D A , generator G A2B Realize the conversion of domain A images to domain B images, discriminator D B Determine whether the input image is a real B-domain image; Generator G B2A Realize the conversion of B domain to A domain image, discriminator D A Determine whether the input image is a real A domain image;

[0028] The generator G A2B , G B2A The network structure is as follows Figure 1As shown in the figure, it consists of three parts: encoder feature extraction, feature conversion and decoder reconstruction feature. The encoder feature extraction part is composed of a padding layer, a 7×7 convolution layer Conv1-1, and two downsampling modules. The padding layer performs a reflection padding operation to fill the image with 3 rows on the top, bottom, left and right. The 7×7 convolution layer Conv1-1 is a convolution layer with 64 7×7 filters, a normalization layer BatchNorm and a Rule activation function. The downsampling module is composed of a convolution layer with 64 3×3 filters, a BatchNorm normalization layer and Relu activation function; the feature conversion part is composed of 9 residual blocks, each of which is composed of two convolutional layers, two normalization layers, a Relu activation function and a jump connection; the decoder reconstruction feature part is composed of two upsampling modules, a 7×7 convolutional layer Conv1-2 and a padding layer, the upsampling module is a deconvolution operation Deconv, the 7×7 convolutional layer Conv1-2 is a combination of a convolutional layer with 64 7×7 filters and a tanh activation function, and the padding layer is a reflection padding operation to fill the image with 3 rows up, down, left and right;

[0029] The discriminator D A , D B like Figure 2 The PatchGAN network structure is shown. First, it passes through a convolution layer consisting of 64 4×4 filters and a convolution layer Conv2-1 consisting of a LeakyRelu activation function, then passes through three feature extraction layers Conv2-2 consisting of a convolution layer, a normalization layer, and a LeakyRelu activation function, and finally passes through a convolution layer with a channel of 1 to obtain the predicted value;

[0030] Step 3. Design regularization loss, train the network, and obtain a network model that implements the style transfer of traditional Chinese ink painting. The specific steps are as follows:

[0031] Step 3.1 Randomly initialize the generator and discriminator network parameters;

[0032] Step 3.2 starts iteratively training the generator and discriminator, and updates the generator and discriminator network parameters; the agreed variable epoch records the number of current network training iterations, epoch is initially 0, and each round of iteration is divided into the following two parts:

[0033] Step 3.2.1: Figure 3 As shown, the generator network parameters are fixed and the discriminator D is adjusted A , D B Network parameters; for the discriminator D A, the real A domain image real_a in the training set trainA is processed by data enhancement using the cutout strategy and recorded as T. The processed image is recorded as real_a_aug and then input into the discriminator D A The output of the network is recorded as pred_real_a_aug; the generator G B2A Generate fake image fake_a and use the cutout strategy for data enhancement. The processed image is recorded as fake_a_aug and input into the discriminator D A The output of the network is recorded as pred_fake_a_aug. The mean square loss function is used to calculate the loss between pred_real_a_aug and the label True, recorded as loss_DA_real, and the loss between pred_fake_a_aug and the label False, recorded as loss_DA_fake. The mean is calculated by combining loss_DA_real and loss_DA_fake to obtain the discriminator D. A Loss loss_D_A; Similarly, for the discriminator D B , the real B domain image real_b in the training set trainB is processed by the cutout strategy for data enhancement. The processed image is recorded as real_b_aug and then input into the discriminator D B The network output is recorded as pred_real_b_aug; the generator G A2B The fake image fake_b is generated by using the cutout strategy for data enhancement. The processed image is recorded as fake_b_aug and then input into the discriminator D B The output of the network is recorded as pred_fake_b_aug. The mean square loss function is used to calculate the loss between pred_real_b_aug and the label True, recorded as loss_DB_real, and the loss between pred_fake_b_aug and the label False, recorded as loss_DB_fake. The mean is calculated by combining loss_DB_real and loss_DB_fake to obtain the discriminator D. B Loss loss_D_B; calculate the discriminator D in this round of iteration A Loss loss_D_A and discriminator D B Loss loss_D_B gradient, back propagation, and re-update the discriminator D through the Adam optimization algorithm A , D B parameter;

[0034] Step 3.2.2 Figure 4 As shown, the discriminator network parameters are fixed and the generator G is adjusted A2B , G B2A Network parameters; for generator GA2B , put the real A domain image real_A into G A2B The network generates a fake image fake_B in domain B, and then inputs fake_B into the generator G. B2A , get the reconstructed A domain image rec_A, define loss_G_A2B variable record generator G A2B Total loss, generator G A2B The total loss consists of four parts: The first part calculates the generator G A2B Ordinary GAN loss, the fake image fake_B is first enhanced to obtain fake_B_aug, and then sent to the discriminator D B The output is obtained, and the mean square loss function is used to calculate the loss between the output and the True label, which is recorded as GANloss_A; the second part uses L1loss to calculate the difference between the reconstructed A-domain image rec_A and the real A-domain image real_A, that is, the cycle consistency loss function, recorded as Cycleloss_A; the third part uses multi-scale structural similarity to measure the similarity between the reconstructed A-domain image rec_A and the real A-domain image real_A in terms of color and hue, etc., recorded as MSSSIM_A, and calculates the 1-MSSSIM_A value as the multi-scale structural similarity loss function MS_SSIMloss_A; the fourth part calculates the sum of the differences between all pixels of the generated fake image fake_B and the adjacent pixels in the vertical and horizontal directions, that is, the total variational loss TVloss_A; the above four loss functions are given appropriate weights and accumulated as the generator G A2B Total loss loss_G_A2B; Similarly, for generating G B2A , the total loss loss_G_B2A is calculated and accumulated by the above four parts; finally, the gradients of loss_G_A2B and loss_G_B2A are calculated and back-propagated, and the generator G is updated again through the Adam optimization algorithm A2B , G B2A Network parameters;

[0035] Step 3.2.3: This round of iteration ends, set epoch = epoch + 1; when epoch reaches the set 200 times, the network stops training and saves the training model model.pt file; otherwise, enter step 3.2 again and repeat the training of the discriminator and generator.

[0036] test:

[0037] The 200 traditional Chinese landscape painting images and 600 natural scenery images in the Chinese_Landscape_Painting2photo dataset are used as test sets, denoted as testA and testB;

[0038] Import the obtained natural scenery to traditional ink painting style transfer model model.pt file into the test code, and input the test set testA and test set testB images into the trained network model G A2B With G B2A In the figure, the migrated result image is obtained, and the SSIM and PSNR evaluation results are calculated.

[0039] Using 6 test images, the peak signal-to-noise ratio (PSNR) and structural similarity (SSIM) of the embodiment of the present invention are compared with the existing style transfer technology algorithm. The results are shown in Table 1.

[0040] Table 1 Comparison of PSNR / SSIM values ​​of each model migration result

[0041]

[0042] The results show that the model constructed by the present invention realizes the migration of unpaired natural scenery photos to traditional ink painting style, and its peak signal-to-noise ratio (PSNR) and structural similarity (SSIM) are higher than the existing technology.

Claims

1. A method for constructing a data-enhanced CycleGAN regularized image style transfer model, characterized in that Follow these steps: Step 1. Collect training set images The traditional Chinese ink landscape images and natural scenery images in the dataset are used as training sets, denoted as trainA and trainB respectively; the traditional Chinese ink landscape images are defined as A domain images, and the natural scenery images are defined as B domain images; Step 2. Define the overall network structure The overall network structure consists of two pairs of generator-discriminator structures, namely G A2B -D B , G B2A -D A , generator G A2B Realize the conversion of domain A images to domain B images, discriminator D B Determine whether the input image is a real B-domain image; Generator G B2A Realize the conversion of B domain to A domain image, discriminator D A Determine whether the input image is a real A domain image; The generator G A2B , G B2A The network structure consists of three parts: encoder feature extraction, feature conversion and decoder reconstruction features. The encoder extracts features and is composed of a padding layer, a 7×7 convolutional layer Conv1-1, and two downsampling modules. The padding layer performs a reflection padding operation to fill the image with 3 rows on the top, bottom, left and right sides. The 7×7 convolutional layer Conv1-1 is a convolutional layer with 64 7×7 filters, a normalization layer BatchNorm and a Rule activation function. The downsampling module is composed of a convolutional layer with 64 3×3 filters, a BatchNorm normalization layer and Relu activation function; the feature conversion part is composed of 9 residual blocks, each of which is composed of two convolutional layers, two normalization layers, a Relu activation function and a jump connection; the decoder reconstruction feature part is composed of two upsampling modules, a 7×7 convolutional layer Conv1-2 and a padding layer, the upsampling module is a deconvolution operation Deconv, the 7×7 convolutional layer Conv1-2 is a combination of a convolutional layer with 64 7×7 filters and a tanh activation function, and the padding layer is a reflection padding operation to fill the image with 3 rows up, down, left and right; The discriminator D A , D B Using the PatchGAN network structure, the network first passes through a convolutional layer consisting of 64 4×4 filters and a convolutional layer Conv2-1 consisting of a LeakyRelu activation function, then passes through three feature extraction layers Conv2-2 consisting of a convolutional layer, a normalization layer, and a LeakyRelu activation function, and finally passes through a convolutional layer with a channel of 1 to obtain the predicted value; Step 3. Design regularization loss, train the network, and obtain a network model that implements the style transfer of traditional Chinese ink painting. The specific steps are as follows: Step 3.1 Randomly initialize the generator and discriminator network parameters; Step 3.2 starts iteratively training the generator and discriminator, and updates the generator and discriminator network parameters; the agreed variable epoch records the number of current network training iterations, epoch is initially 0, and each round of iteration is divided into the following two parts: Step 3.2.1: Fix the generator network parameters and adjust the discriminator D A , D B Network parameters; for the discriminator D A , the real A domain image real_a in the training set trainA is processed by data enhancement using the cutout strategy and recorded as T. The processed image is recorded as real_a_aug and then input into the discriminator D A The output of the network is recorded as pred_real_a_aug; the generator G B2A Generate fake image fake_a and use the cutout strategy for data enhancement. The processed image is recorded as fake_a_aug and input into the discriminator D A The output of the network is recorded as pred_fake_a_aug. The mean square loss function is used to calculate the loss between pred_real_a_aug and the label True, recorded as loss_DA_real, and the loss between pred_fake_a_aug and the label False, recorded as loss_DA_fake. The mean is calculated by combining loss_DA_real and loss_DA_fake to obtain the discriminator D. A loss_D_A; Similarly, for the discriminator D B , the real B domain image real_b in the training set trainB is processed by the cutout strategy for data enhancement. The processed image is recorded as real_b_aug and then input into the discriminator D B The network output is recorded as pred_real_b_aug; the generator G A2B The fake image fake_b is generated by using the cutout strategy for data enhancement. The processed image is recorded as fake_b_aug and then input into the discriminator D B The output of the network is recorded as pred_fake_b_aug. The mean square loss function is used to calculate the loss between pred_real_b_aug and the label True, recorded as loss_DB_real, and the loss between pred_fake_b_aug and the label False, recorded as loss_DB_fake. The mean is calculated by combining loss_DB_real and loss_DB_fake to obtain the discriminator D. B Loss loss_D_B; calculate the discriminator D in this round of iteration A Loss loss_D_A and discriminator D B Loss loss_D_B gradient, back propagation, and re-update the discriminator D through the Adam optimization algorithm A , D B parameter; Step 3.2.2 Fix the discriminator network parameters and adjust the generator G A2B , G B2A Network parameters; for generator G A2B , put the real A domain image real_A into G A2B The network generates a fake image fake_B in domain B, and then inputs fake_B into the generator G. B2A , get the reconstructed A domain image rec_A, define loss_G_A2B variable record generator G A2B Total loss, generator G A2B The total loss consists of four parts: The first part calculates the generator G A2B Ordinary GAN loss, the fake image fake_B is first enhanced to obtain fake_B_aug, and then sent to the discriminator D B The output is obtained, and the mean square loss function is used to calculate the loss between the output and the True label, which is recorded as GANloss_A; the second part uses L1loss to calculate the difference between the reconstructed A-domain image rec_A and the real A-domain image real_A, that is, the cycle consistency loss function, recorded as Cycleloss_A; the third part uses multi-scale structural similarity to measure the similarity between the reconstructed A-domain image rec_A and the real A-domain image real_A in terms of color and hue, etc., recorded as MSSSIM_A, and calculates the 1-MSSSIM_A value as the multi-scale structural similarity loss function MS_SSIMloss_A; the fourth part calculates the sum of the differences between all pixels of the generated fake image fake_B and the adjacent pixels in the vertical and horizontal directions, that is, the total variational loss TVloss_A; the above four loss functions are weighted and accumulated as the generator G A2B Total loss loss_G_A2B; Similarly, for generating G B2A , the total loss loss_G_B2A is calculated and accumulated by the above four parts; finally, the gradients of loss_G_A2B and loss_G_B2A are calculated and back-propagated, and the generator G is updated again through the Adam optimization algorithm A2B , G B2A Network parameters; Step 3.2.3: This round of iteration ends, set epoch = epoch + 1; when the epoch reaches the set number of times, the network stops training and saves the training model model.pt file; otherwise, enter step 3.2 again and repeat the training of the discriminator and generator.

Images