Semi-supervised image translation method based on generative adversarial network

Abstract

The invention provides a semi-supervised image translation method based on a generative adversarial network. According to the method, a source domain image is converted into a target domain image through two stages of encoding and decoding of a generator. In the encoding stage, the network extracts feature representation from a source domain image; in a decoding stage, a corresponding target domain image is generated according to the extracted features. In order to improve the feature transmission efficiency, the middle features in the encoding stage are directly transmitted to a decoding module through jump connection in the decoding stage, so that the reconstruction effect of the image is enhanced. Meanwhile, a CBAM (attention mechanism module) is introduced into the decoding module, so that key feature information in the image can be accurately captured, and irrelevant information which is relatively weak in generation relation with the target image is inhibited. The mechanism helps the network to focus on valuable features, thereby improving the quality and accuracy of image translation.

Description

Technical Field

[0001] The present invention belongs to the technical field of image processing, and in particular relates to a semi-supervised image translation method based on a generative adversarial network. Background Art

[0002] Image translation is an important research direction of deep learning in the field of computer vision. It aims to convert images from one field to another field by adjusting the image's color, texture and other features while retaining the image's posture and general outline. In short, image translation involves converting features between different visual domains while maintaining the structural information of the original image to the greatest extent possible. Many classic computer vision problems, such as image style transfer, image dehazing, and super-resolution reconstruction, can be regarded as applications of image translation. Although existing image translation methods have made progress to a certain extent, there are still problems with insufficient accuracy and residual redundant information, resulting in less than ideal translation results. Summary of the invention

[0003] In order to solve the above problems, the present invention provides a semi-supervised image translation method based on a generative adversarial network, comprising the following steps:

[0004] Step S1: For the translation task, obtain a dataset with paired images and determine the source domain image and the target domain image.

[0005] Step S2: preprocessing the acquired paired image data set.

[0006] Step S3: Construct an image translation model based on generative adversarial network and attention mechanism.

[0007] Step S4: training a weak image translation model based on the preprocessed paired image dataset.

[0008] Step S5: Input the source domain image into the trained image translation model for testing to obtain a translated target domain image.

[0009] Furthermore, the preprocessing method in step S2 is to process the image into a size of 256×256.

[0010] Furthermore, step S3 includes the following sub-steps:

[0011] Sub-step S31: The image translation model includes two image generators G AB and G BA and two discriminators D A and D B。

[0012] Sub-step S32: Use LSGAN adversarial loss to replace the original adversarial loss function to make the training process more stable.

[0013] Sub-step S33: Generator G AB and G BA It is divided into encoding stage and decoding stage. In the decoding stage, the CBAM attention mechanism is used to capture the important features of the source domain image.

[0014] Furthermore, the generator G AB Including anti-loss , cycle consistency loss , VGG feature block loss and VGG feature block style loss , the specific function is: {\mathcal{L}}_{{G}_{AB\_ adv}}={E}_{x\sim represents the mathematical expectation, represents the image in image domain A, y represents the image in image domain B, represents the feature map extracted using the VGG-19 pre-trained model, where, The values ​​are 3, 4, and 5, corresponding to the relu3-1, relu4-1, and relu5-1 layers of VGG-19, respectively. represents the jth 3×3 feature block divided by the input, Indicates The maximum number of feature blocks of size 3×3 that the layer VGG-19 features can be divided into, is the Gram matrix of the input.

[0015] Furthermore, the generator G AB The total loss is:

[0016] For the generator G BA Adversarial loss: {\mathcal{L}}_{{G}_{BA\_ adv}}={E}_{x\sim

[0017] is the discriminator D A Adversarial loss: {\mathcal{L}}_{{D}_{A\_ adv}}={E}_{x\sim {X}_{A}}\left [ {\left ( {{D}_{A}\left ( {x} \right )-1} \right )^{2}} \right ]\left [ \left ( {} \right ) \right ]

[0018] is the discriminator D B Adversarial loss: {\mathcal{L}}_{{D}_{B\_ adv}}={E}_{x\sim {{D}_{B}\left ( {y} \right )-1} \right )^{2}} \right ]\left [ \left ( {} \right ) \right ]

[0019] The total loss of the further proposed image translation model is: BRIEF DESCRIPTION OF THE DRAWINGS

[0020] Figure 1 This is a flowchart of the semi-supervised image translation method according to the present invention.

[0021] Figure 2 This is a model structure diagram of the semi-supervised image translation method of the present invention.

[0022] Figure 3 This is a structural diagram of the decoding layer of the generator in the semi-supervised image translation method of the present invention.

[0023] Figure 4 This is a comparison diagram of the effects of the semi-supervised image translation method of the present invention and other image translation methods; DETAILED DESCRIPTION

[0024] The specific embodiments of the present invention are described in detail below in conjunction with the accompanying drawings so that those skilled in the art can understand the present invention.

[0025] like Figure 1 As shown, a semi-supervised image translation method based on a generative adversarial network includes at least the following steps:

[0026] Step S1: For the translation task, obtain a dataset with paired images and determine the source domain image and the target domain image.

[0027] Step S2: preprocessing the acquired paired image data set.

[0028] Step S3: Construct an image translation model based on generative adversarial network and attention mechanism.

[0029] Step S4: training a weak image translation model based on the preprocessed paired image dataset.

[0030] Step S5: Input the source domain image into the trained image translation model for testing to obtain a translated target domain image.

[0031] like Figure 2 As shown, the image in the A image domain is input to the generator network G AB , generate an image with the style of the B image domain. The generated image is input to the generator network G BA In the above example, we can generate images with the style of image domain A. The discriminator D B Determine the authenticity of the generated image in the B image domain, the discriminator D A Determine the authenticity of the generated image in the A image domain. In order to train the image translation model, we use the VGG feature block loss, VGG feature block style loss and cycle consistency loss in addition to the adversarial loss obtained by the discriminator.

[0032] For the input image, transform it to 256x256x3 size. The generator is divided into encoding stage and decoding stage. First, the residual block of convolution with 64 channels, 3 convolution kernels, 1 stride, and 1 padding is used. The second, third, fourth and fifth layers use the residual block of convolution with 128 channels, 3 convolution kernels, 2 stride, and 1 padding. The sixth layer consists of nine residual blocks of convolution with 128 channels, 3 convolution kernels, 1 stride, and 1 padding. Figure 3 As shown in the figure, it is a decoding layer structure in the decoding stage. The decoding layer accepts the input of the encoding layer feature stream and the decoding layer feature stream and stacks them according to the channel dimension, and then inputs them into the CBAM attention mechanism. The output is upsampled using the nearest neighbor interpolation algorithm, and then input into the subsequent convolution residual block. The upsampling operation uses the nearest neighbor interpolation algorithm. In the decoding stage, the seventh layer takes the output of the sixth and fifth layers as input, the eighth layer takes the output of the seventh and fourth layers as input, the ninth layer takes the output of the eighth and third layers as input, the tenth layer takes the output of the ninth and second layers as input, and the eleventh layer takes the output of the tenth and first layers as input. The eleventh layer does not perform upsampling. The convolution kernel size used for all convolution operations in the decoding stage is 3, and the step size is 1. The final output is an output image of size 256x256x3.

[0033] For the discriminator network, a multi-scale discriminator network with four different scales is used, and the average pooling layer with a kernel size of 3 and a step size of 3 is used to obtain features of different scales. The input is 256x256x3 image data, and the results obtained by the discriminators of different scales are uniformly adjusted to 30x30 using the nearest neighbor interpolation algorithm. The final result obtained by the multi-scale discriminator network is a feature of size 30x30x4.

[0034] After the model parameters are trained, the image belonging to the image domain A is passed to the generator G AB , and obtain the translated target domain image.

[0035] For the generator G AB The adversarial loss is: {\mathcal{L}}_{{G}_{AB\_ adv}}={E}_{x\sim

[0036] For the generator G ABThe cycle consistency loss of is:

[0037] To generate the VGG feature block loss of the image, To generate the VGG feature block style loss of the image, the specific function is: represents the mathematical expectation, represents the image in image domain A, y represents the image in image domain B, represents the feature map extracted using the VGG-19 pre-trained model, where, The values ​​are 3, 4, and 5, corresponding to the relu3-1, relu4-1, and relu5-1 layers of VGG-19, respectively. represents the jth 3×3 feature block divided by the input, Indicates The maximum number of feature blocks of size 3×3 that the layer VGG-19 features can be divided into, is the Gram matrix of the input.

[0038] Generator G AB The total loss of is the sum of adversarial loss, cycle consistency loss, VGG feature block loss and VGG feature block style loss. Therefore, the generator G AB The total loss is:

[0039] For the generator G BA The adversarial loss is: {\mathcal{L}}_{{G}_{BA\_ adv}}={E}_{x\sim

[0040] is the discriminator D A The adversarial loss is: {\mathcal{L}}_{{D}_{A\_ adv}}={E}_{x\sim {X}_{A}}\left [ {\left ( {{D}_{A}\left ( {x} \right )-1} \right )^{2}} \right ]\left [ \left ( {} \right ) \right ]

[0041] is the discriminator D B The adversarial loss is: {\mathcal{L}}_{{D}_{B\_ adv}}={E}_{x\sim {{D}_{B}\left ( {y} \right )-1} \right )^{2}} \right ]\left [ \left ( {} \right ) \right ] This implementation uses a publicly available paired image dataset, the details of which are as follows: The WildSketch dataset contains 396 pairs of real face photos and face sketch images for training and 400 pairs of real face photos and face sketch images for testing.

[0042] In order to verify the superiority of this implementation (Ours), this implementation is compared with several existing image translation methods, including CycleGAN, CHAN, and SCG methods. The performance of these methods in the face sketch images generated from the above public datasets based on real face images in terms of FSIM, LPIPS, and DISTS indicators will be compared. The specific data comparison is shown in Table 1. Figure 4 This is a comparison chart of the effects of this implementation method and the CycleGAN, CHAN, and SCG methods.

[0043] Table Quantitative results comparison of face photo to sketch image conversion FSIM LPIPS DISTS CycleGAN 0.6610 0.3553 0.2602 CHAN 0.6675 0.3399 0.2463 SCG 0.7010 0.3165 0.2529 Ours 0.7056 0.2931 0.2360 The larger the value of FSIM (Feature Similarity Index), the better the image quality. The smaller the value of LPIPS (Learning Perceptual Patch Similarity), the smaller the visual difference and the higher the quality. The smaller the value of DISTS (Deep Image Structural Similarity), the higher the similarity between images. By comparing the data in the above table, we can see that Ours has achieved the best performance in all three evaluation indicators, indicating that the images generated by our method are most similar to the real images. The quantitative results fully demonstrate the superiority of Ours.

Claims

1. A semi-supervised image translation method based on generative adversarial network, characterized in that: The steps include: For translation tasks, obtain a dataset with paired images and determine the source domain images and the target domain images; Preprocessing the acquired paired image dataset; Build a semi-supervised image translation model based on generative adversarial networks; Based on the preprocessed paired image dataset, a semi-supervised image translation model is trained; The source domain image is input into the trained image translation model for testing to obtain the translated target domain image.

2. According to claim 1, a semi-supervised image translation method based on generative adversarial network is characterized in that: Divide the dataset into training and testing sets.

3. The semi-supervised image translation method based on generative adversarial network according to claim 1, characterized in that: The images in the dataset are processed to 256×256 size.

4. The semi-supervised image translation method based on generative adversarial network according to claim 1, characterized in that: The image translation model consists of two image generators G AB and G BA and two discriminators D A and D B .

5. The semi-supervised image translation method based on generative adversarial network according to claim 4, characterized in that: Generator G AB and G BA It is divided into encoding stage and decoding stage. In the decoding stage, the CBAM attention mechanism is used to capture the important features of the source domain image.

6. The semi-supervised image translation method based on generative adversarial network according to claim 4, characterized in that: Generator G AB Including anti-loss , cycle consistency loss , VGG feature block loss and VGG feature block style loss Four parts of loss, the specific function is: represents the mathematical expectation, represents the image in image domain A, y represents the image in image domain B, represents the feature map extracted using the VGG-19 pre-trained model, where, The values ​​are 3, 4, and 5, corresponding to the relu3-1, relu4-1, and relu5-1 layers of VGG-19, respectively. represents the jth 3×3 feature block divided by the input, Indicates The maximum number of feature blocks of size 3×3 that the layer VGG-19 features can be divided into, is the Gram matrix of the input; The generator G AB The total loss is:

7. The semi-supervised image translation method based on generative adversarial network according to claim 4, characterized in that: Generator G BA Loss against loss , the specific function is:

8. The semi-supervised image translation method based on generative adversarial network according to claim 4, characterized in that: is the discriminator D A The adversarial loss, the specific function is:

9. The semi-supervised image translation method based on generative adversarial network according to claim 4, characterized in that: is the discriminator D B The adversarial loss, the specific function is:

10. The semi-supervised image translation method based on generative adversarial network according to claim 4, characterized in that: The total loss of the image translation model is:

Images