A Generative Adversarial Network-Based Image Translation Method with Dual Encoders

Abstract

This invention proposes an image translation method based on a dual-encoder generative adversarial network (GAN), comprising: constructing a dual-encoder GAN model, wherein the dual-encoder GAN model includes a generator G, a discriminator D, and an encoder E, and the discriminator D includes an encoding part D. E And classification section D C Sample images are acquired from the source domain and input into the encoding parts D of encoder E and discriminator D, respectively. E The first and second encoded features are obtained; these features are then concatenated and input into the generator G to obtain the translated image. The input image is encoded using the encoding parts of the encoder and discriminator, and the resulting encoded features are concatenated. To better combine the two encodings, a novel normalization method, Adaptive Multi-Feature Layer Normalization (AdaMFLN), is proposed. This method enables better decoupling and translation of the image.

Description

Technical Field

[0001] This invention belongs to the technical field of image translation, specifically relating to an image translation method based on a dual-encoder generative adversarial network. Background Technology

[0002] Image-to-image translation aims to learn a function that maps an image to two different domains. This unique learning approach has given rise to many visual tasks, such as image inpainting, super-resolution, colorization, and style transfer, thus attracting widespread attention from researchers in machine learning and computer vision. When using paired datasets for translation, supervised image translation can be performed based on conditional adversarial networks (CANN) for image-to-image transformation. However, in real-world scenarios, large amounts of paired data are often unavailable for training. Instead, image translation models with only unpaired data from two domains and a limited number of target domain samples often achieve excellent results in unsupervised image translation tasks.

[0003] Despite achieving promising results in unsupervised tasks, current work tightly encodes images in a single way. When we re-examine successful translation frameworks (such as CycleGAN), most consist of three components per domain: an encoder that embeds the input image into a low-dimensional hidden space, a generator that transforms the hidden vectors into images of other domains, and a discriminator trained using GAN for domain alignment. The paper "Reusing Discriminators for Encoding Towards Unsupervised Image-to-Image Translation. 2020 IEEE / CVFConference on Computer Vision and Pattern Recognition (CVPR)" redesigns these components, abandoning the traditional encoder and replacing it with the encoding part of the discriminator, simplifying the network and improving translation performance. However, these network structures only consider the single role of the encoder. In unsupervised image translation tasks, discriminators and generators play different roles in encoding images. For example, the discriminator's image processing can be divided into encoding and classification. DCGAN reveals the encoding capability of the discriminator: the first 6 convolutional features learned from the last convolutional layer of the discriminator can be observed to have a strong response to the input image, while the encoder in the generator is responsible for transferring between the two domains.

[0004] Based on the above motivation, this application proposes a generative adversarial network image translation method based on dual encoders, which can fuse the two encoding results to achieve better translation results. Summary of the Invention

[0005] To address the problems of existing network structures that only consider the single role of the encoder and the difficulty in easily connecting the features obtained from two encoders, this application provides a generative adversarial network image translation method based on dual encoders to solve the aforementioned technical deficiencies.

[0006] According to one aspect of the present invention, a generative adversarial network-based image translation method with dual encoders is proposed, the method comprising the following steps:

[0007] S1. Construct a dual-encoder generative adversarial network (GAN) model. The dual-encoder GAN model includes a generator G, a discriminator D, and an encoder E, where the discriminator D includes an encoding part D. E And classification section D C ;

[0008] S2. Acquire sample images from the source domain and input the sample images into the encoding parts D of encoder E and discriminator D, respectively. E Obtain the first coding feature and the second coding feature; and

[0009] S3. The first and second coding features obtained in step S2 are concatenated, and the concatenated features are input into the generator G to finally obtain the translated image.

[0010] In the aforementioned technical solution, the discriminator D can learn the differences between real and generated images, while the generator G can learn the differences between images in different domains. Therefore, the features obtained by the dual-encoder generative adversarial network model constructed in this application can simultaneously satisfy both of the above characteristics, resulting in better translation. Moreover, since the two encodings have different learning focuses, this method achieves better decoupling, and changing different features can realize a "multimodal" image translation task.

[0011] In a specific embodiment, in step S3, the first and second encoded features obtained in step S2 are concatenated using the Adaptive Multi-Feature Layer Normalization (AdaMFLN) method. The Adaptive Multi-Feature Layer Normalization (AdaMFLN) method combines different features and adaptively performs layer normalization operations.

[0012] An adaptive multi-feature layer normalization method is proposed, which maps different encoded features to the same dimension to better fit the dual encoder generative adversarial network model structure constructed in this application.

[0013] In a specific embodiment, in step S3, the parameters of the Adaptive Multi-Feature Layer Normalization Method (AdaMFLN) are dynamically calculated by the fully connected layer of the encoded feature map:

[0014]

[0015] In the formula, x G and x D These represent the encodings obtained by the generator G and the discriminator D, respectively. x G and x D The layer mean, x G and x D The layer standard deviation; γ and β are parameters generated by fully connected layers; specifically, x G and x D First, the AdaptiveAvgPool operation is performed separately to concatenate the obtained features, and then γ and β are obtained through two fully connected layers.

[0016] The changes in the above parameters are used to indicate which normalization method the dual-encoder generative adversarial network model constructed in this application prefers.

[0017] In a specific embodiment, in step S1, the residual module of generator G incorporates the Adaptive Multi-Feature Layer Normalization (AdaMFLN) method, and the upsampling layer and convolutional layer of generator G incorporate the Adaptive Multi-Feature Layer Normalization (AdaMFLN) method.

[0018] The above technical solutions make it easier to control the style during the translation process.

[0019] In a specific embodiment, in step S1, the generator G uses ReLU as the activation function, and the discriminator D uses leak-ReLU with a slope of 0.2 as the activation function.

[0020] In a specific embodiment, in step S1, the loss function expression of the dual-encoder generative adversarial network model is:

[0021] Loss=λ1L gan +λ2L cycle +λ3L recon

[0022] Among them, L gan L represents the adversarial loss function; cycle L represents the cycle-consistent loss function; recon λ1, λ2, and λ3 represent the reconstruction loss function; λ1, λ2, and λ3 are hyperparameters that control the weights of each loss function.

[0023] The adversarial loss function aims to achieve domain transfer, while the reconstruction loss function and periodic consistency loss function are designed to address the unrecognizable nature of image translation. Since the encoder is composed of the encoding parts of both the encoder and discriminator, the training method of the proposed dual-encoder generative adversarial network model differs from that of ordinary image translation models. Specifically, the encoder and generator are trained simultaneously, while the encoding part of the discriminator is trained together with the discriminator.

[0024] In a specific embodiment, the expression for the adversarial loss function is:

[0025]

[0026] Where X is the source domain formed by the images, and Y is the target domain formed by the images; x is the image sampled from the source domain, and y is the image sampled from the target domain; G x→y E represents the result of transforming an image x from the source domain into the target domain using a generator G; y～Y This indicates that image y originates from the target domain Y, E x～X This indicates that image x originates from source domain X; D y (y) represents the discriminant's judgment result for y; D y (G x→y (E(x)⊙D E (x))) represents the discriminator's judgment on G x→y (E(x)⊙D E The discrimination result of (x)); E(x) represents the encoding result of encoder E on x; D E (x) represents the coding part D of the discriminator D. E For the encoding result of x, ⊙ represents the concatenation operation.

[0027] This application utilizes least-squares adversarial loss to make training more stable and generate higher-quality results.

[0028] In a specific embodiment, the expression for the cycle-consistent loss function is:

[0029]

[0030] Where x is the image sampled from the source domain; G x→y This represents the result of transforming an image x from the source domain into the target domain using a generator G; G y→x This represents the result of transforming the image y from the target domain into the source domain using generator G; Encoder E y right The encoding result; This represents the encoded part of the discriminator D. The encoding result of x; This represents the encoded part of the discriminator D. right The encoding result; ⊙ represents the concatenation operation; |·| represents L1 regular expression.

[0031] A cycle consistency loss function is introduced to make the two generators invertible.

[0032] In a specific embodiment, the expression for the reconstruction loss function is:

[0033]

[0034] Among them, E x～X Indicates that image x comes from source domain X; x is an image sampled from the source domain; G y→x This represents the result of transforming the image y from the target domain into the source domain using generator G; D E (x) represents the coding part D of the discriminator D. E The encoding result of x; ⊙ represents the concatenation operation; |·| represents L1 regularization.

[0035] Approximating the generator to the identity function is another key control technique in CycleGAN. Unlike CycleGAN's identity loss, which is based on the domain similarity assumption, reconstruction is based on the shared latent space assumption. The reconstruction loss is a translation regularization that approximates the identity mapping when the hidden vectors of the real samples in the source domain are used as input to the source domain generator.

[0036] According to another aspect of the present invention, a computer-readable storage medium is provided, wherein a computer program is stored therein, and when the computer program is executed by a processor, the method described in any of the above is implemented.

[0037] Compared with the prior art, the beneficial results of the present invention are as follows:

[0038] (1) This invention proposes a new unsupervised image translation model. The model uses the encoding parts of the encoder and discriminator to encode the image, and then concatenates the obtained features to perform the image translation task. This method can retain the learning characteristics of the discriminator and the encoder at the same time, and can learn the differences between different domains as well as the differences between real data and generated data.

[0039] (2) This invention proposes a new normalization method, the Adaptive Multi-Feature Layer Normalization Method (AdaILGN), which combines different features and adaptively performs layer normalization operations.

[0040] (3) This invention achieves better decoupling of the generative adversarial network-based image translation model. By combining different coding features, a "multimodal" image translation task can be achieved, which is something that other single-modal image translation models cannot do. Attached Figure Description

[0041] The accompanying drawings are included to provide a further understanding of the embodiments and are incorporated in and constitute a part of this specification. The drawings illustrate embodiments and, together with the description, serve to explain the principles of the invention. Other embodiments and many anticipated advantages of the embodiments will be readily recognized as they become better understood through reference to the following detailed description. Other features, objects, and advantages of this application will become more apparent from reading the following detailed description of non-limiting embodiments with reference to the accompanying drawings:

[0042] Figure 1 This is a flowchart of an image translation method based on a dual-encoder generative adversarial network according to the present invention;

[0043] Figure 2 This is an architecture diagram of the generative adversarial network model based on dual encoders of the present invention;

[0044] Figure 3 This is a comparison diagram between traditional image translation methods and the model proposed in this application;

[0045] Figure 4 This is a comparison chart of the translation results of existing image translation models and the model proposed in this application;

[0046] Figure 5 This is a flowchart illustrating the operation of controlling different styles of an image through different encoding features according to a specific embodiment of the present invention;

[0047] Figure 6 This is according to a specific embodiment of the present invention. A diagram illustrating the impact of different encoding features on style in a dataset. Detailed Implementation

[0048] The present application will now be described in further detail with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the invention and not intended to limit it. Furthermore, it should be noted that, for ease of description, only the parts relevant to the invention are shown in the accompanying drawings.

[0049] Figure 1 A flowchart of a generative adversarial network-based image translation method according to the present invention is shown, as follows: Figure 1 As shown, the method includes:

[0050] S1. Construct a dual-encoder generative adversarial network (GAN) model. The dual-encoder GAN model includes a generator G, a discriminator D, and an encoder E, where the discriminator D includes an encoding part D. E And classification section D C ;

[0051] S2. Acquire sample images from the source domain and input the sample images into the encoding parts D of encoder E and discriminator D, respectively. E Obtain the first coding feature and the second coding feature; and

[0052] S3. The first and second coding features obtained in step S2 are concatenated, and the concatenated features are input into the generator G to finally obtain the translated image.

[0053] Figure 2 The architecture diagram of the generative adversarial network model based on dual encoders of the present invention is shown. Figure 2 As shown, the goal of this application is to train a function G. s→t The function G s→t The image is extracted from the source domain X using only unpaired samples extracted from each domain. s Mapping to target domain X t The generative adversarial network model framework in this application consists of two generators G. s→t and G t→s Two discriminators D s and D t With two encoders E s and E t The system consists of a discriminator, which is further divided into an encoding part D. E And classification section D C ,Right now

[0054] In a specific embodiment, let x∈{X} s X t} represents a pair of samples collected from the source and target domains. The dual-encoder generative adversarial network model provided in this application includes an independent encoder E. s A discriminator D s The discriminator consists of two parts: the encoding part and the coding part. and classification section Model receives X s As input, input respectively With E s To obtain the encoded features, i.e. The obtained encoded features are then concatenated, i.e. Where ⊙ represents the concat operation, the concatenated features are then input into the generator G. s→t In this process, the translated image is obtained, namely x.t =G s→t (h x (x s Inspired by AdaILN, this application also proposes an adaptive multi-feature layer normalization method (AdaMFLN), whose parameters are dynamically calculated by the fully connected layers encoding the feature maps:

[0055]

[0056] x G and x D These represent the encodings obtained from the image after passing through the generator and discriminator, respectively. and x G and x D The layer mean and layer standard deviation. γ and β are parameters generated by fully connected layers. Specifically, x G and x D First, the AdaptiveAvgPool operation is performed separately to concatenate the obtained features, and then γ and β are obtained through two fully connected layers.

[0057] In addition, unlike traditional methods, this application not only adds an adaptive multi-feature layer normalization method (AdaMFLN) to the residual module, but also adds an adaptive multi-feature layer normalization method (AdaMFLN) after the upsampling layer and the convolutional layer. Experiments show that this method can more easily control the style in the translation process, as demonstrated in the experimental records below.

[0058] In the discriminator module, let x∈{X} t G s→t (X s The discriminator represents the sampling of the real image in the target domain and the generated image of the target domain after translation. As mentioned above, the discriminator consists of an encoding part and a classification part. The classification part of the discriminator uses a multi-scale classification module. They performed judgments on the encoded features at different scales, and combined with the encoding part of the discriminator, the final output D of the discriminator was obtained. s (x s ) evolved into

[0059] In a specific embodiment, function G s→tThe training process involves three types of losses: adversarial loss, identity reconstruction loss, and periodic consistency loss. Adversarial loss aims to achieve domain transfer, while reconstruction loss and periodic consistency loss address the unrecognizable nature of image translation. Since the encoder is composed of the encoder and discriminator's encoding parts, the training method for the dual-encoder generative adversarial network model differs from ordinary image translation models. Specifically, in the dual-encoder generative adversarial network model, the encoder and generator are trained simultaneously, while the discriminator's encoding part is trained together with the discriminator.

[0060] The loss function for the dual-encoder generative adversarial network model is as follows:

[0061] (1) Adversarial loss function:

[0062] This application utilizes least-squares adversarial loss to make training more stable and generate higher-quality data. The specific implementation is shown below:

[0063]

[0064] (2) Cyclic consistent loss function:

[0065] CycleGAN and Discoegan first introduced cycle consistency loss, making the two generators reversible.

[0066]

[0067] Where |·| represents l1 regularity.

[0068] (3) Reconstruction loss function:

[0069] The generator's approximation of the identity function is another key control technique in CycleGAN. Unlike CycleGAN's identity loss, which is based on the domain similarity assumption, reconstruction is based on the shared latent space assumption. The reconstruction loss, when using the hidden vectors of the source domain's true samples as input to the source domain generator, performs a translation regularization to approximate the identity mapping. The specific reconstruction loss function is as follows:

[0070]

[0071] Similarly, the translation loss from domain Y to domain X is denoted as

[0072] The final total loss of the dual-encoder generative adversarial network model is shown below:

[0073] Loss=λ1L gan +λ2L cycle +λ3L recon

[0074] Where λ1, λ2, and λ3 are hyperparameters that control the weights of each loss. In the model proposed in this application, they are fixed as λ1 = 1, λ2 = 10, and λ3 = 10.

[0075] Figure 3 A comparison diagram is shown between traditional image translation methods and the model proposed in this application. (See diagram for reference.) Figure 3 As shown, Figure 3 The top (cyclegan) shows a traditional image translation model that uses a separate encoder for encoding. The bottom shows the image translation model proposed in this application, which uses both the encoder and the discriminator's encoding parts for encoding.

[0076] The proposed scheme has the following advantages: 1. The discriminator can learn the differences between the real image and the generated image, while the generator can learn the differences between images in different domains. Therefore, the features obtained by this model can simultaneously satisfy both of these characteristics, resulting in better translation. 2. Since the two encodings have different learning focuses, this method achieves better decoupling. Through experiments, we found that changing different features can achieve a "multimodal" image translation task.

[0077] The following experiments were conducted to verify the generative adversarial network image translation method based on dual encoders proposed in this application.

[0078] In this embodiment of the invention, the dual-encoder generative adversarial network model is compared with four current state-of-the-art models (NICE-GAN, U-GAT-IT, CycleGAN, and MUNIT) for unsupervised image translation. All models are tested using publicly available code provided by the inventors on GitHub.

[0079] The experiments were conducted on four commonly used unpaired image benchmarks: and The first three datasets were provided by CycleGAN and were divided into: 1067 / 120 (horses), 1334 / 140 (zebras), 1231 / 309 (summer), 962 / 238 (winter), 400 / 400 (Van Gogh), and 6287 / 751 (photos). The last dataset was studied in DRIT, with training and testing sets of: 771 / 100 (cats) and 1264 / 100 (dogs). All images in all datasets were cropped and resized to 256×256 for both training and testing.

[0080] In this embodiment of the invention, ReLU is used as the activation function in the generator, and Leak-ReLU with a slope of 0.2 is used as the activation function in the discriminator. All models are trained on an NVIDIA RTX 2080Ti GPU using an Adam optimizer with a learning rate of 0.0001 and (β1, β2) = (0.5, 0.999). For data augmentation, images are horizontally flipped with a probability of 0.5, resized to 286×286, and randomly cropped to 256×256 pixels. The batch size for all experiments is set to 1. Weight decay is used at a rate of 0.0001, and all models are trained for more than 100K iterations.

[0081] The table below shows the translation comparison results between existing image translation models and the model proposed in this application (FID and KID×100 are used for different algorithms, and all methods are trained to the 100kth iteration):

[0082]

[0083]

[0084] The FID in the table is calculated by comparing the statistical data of the generated samples with the real samples. The FID fits a Gaussian distribution to the hidden activations of each set of compared images, and then the Frechet distance (also known as the Wasserstein-2 distance) is calculated. The lower the FID, the closer the generated image is to the real one.

[0085] KID is a metric similar to FID, but it doesn't require the normal distribution assumption of FID; it's an unbiased estimate. KID is essentially the squared multinomial kernel function MMD in the Inception feature representation space. A lower KID indicates that the real and generated images are visually more similar. The KID implementation is based on https: / / github.com / mbinkowski / MMD-GAN, where the hidden representations come from the pool3 layer of Inception-v3.

[0086] As shown in the table above, except for the cat→dog and zebra→horse datasets, the dual-encoder generative adversarial network model proposed in this invention achieved the best scores in both FID and KID, indicating that the dual-encoder generative adversarial network model proposed in this invention has good image translation capabilities in different datasets.

[0087] The table below shows the impact of each encoder in the model and the proposed normalization method on the overall model:

[0088]

[0089]

[0090] The table analyzes the impact of three components (the ordinary encoder, the encoder in the discriminator, and adaptive multi-feature layer normalization) on the entire dual-encoder generative adversarial network model. As shown in the table, using both the encoder and discriminator's encoding parts simultaneously significantly reduces FID and KID scores. Furthermore, the adaptive multi-feature layer normalization method (AdaMFLN) not only reduces FID and KID scores but also better enables the model to control patterns.

[0091] Figure 5 A flowchart illustrating different style operations of an image through different encoding features, according to a specific embodiment of the present invention, is shown. Figure 5 As shown, the dual-encoder generative adversarial network model encodes the input image using two encoders. To demonstrate the effectiveness of this decoupling method, experiments were conducted using different sampled input images for encoding. The resulting codes were then subjected to average pooling, and the resulting style control codes were input into the subsequent generator. The experiments proved that different codes can successfully control different styles of the image.

[0092] Figure 6 A specific embodiment of the present invention is shown. A diagram illustrating the impact of different encoding features on style in a dataset. Figure 6 It can be seen that this decoupling method can effectively control the style of the image, realizing a "multimodal" image translation task.

[0093] This application proposes a generative adversarial network (GAN) image translation method based on a dual-encoder architecture. The dual-encoder GAN model encodes the input image using the encoding parts of both the encoder and discriminator. The resulting encoded features are then concatenated, and a novel normalization method (Adaptive Multi-Feature Layer Normalization (AdaMFLN)) is proposed. This approach enables better decoupling and translation of images, and the translated images can be applied to commercial applications such as film, advertising, games, and even virtual reality. Furthermore, it can be used for automatic translation of faces or objects. This efficient image translation model can save significant costs and create more interesting applications.

[0094] The above description is merely a preferred embodiment of this application and an explanation of the technical principles employed. Those skilled in the art should understand that the scope of the invention involved in this application is not limited to technical solutions formed by specific combinations of the above-described technical features, but should also cover other technical solutions formed by arbitrary combinations of the above-described technical features or their equivalents without departing from the above-described inventive concept. For example, technical solutions formed by substituting the above features with (but not limited to) technical features with similar functions disclosed in this application.

Claims

1. A dual-encoder based generative adversarial network image translation method, characterized in that, The method comprises the following steps: S1, constructing a dual-encoder generative adversarial network model, the dual-encoder generative adversarial network model comprising a generator G, a discriminator D and an encoder E, wherein the discriminator D comprises an encoding part and a classification part ; S2, collecting a sample image from the source domain, inputting the sample image into an encoding part of the encoder E and the discriminator D respectively , obtaining a first encoding feature and a second encoding feature; and S3, the first encoding feature and the second encoding feature obtained in step S2 are spliced, and the spliced features are input into the generator G to finally obtain a translated image; In step S3, the first encoding feature and the second encoding feature obtained in step S2 are spliced by using an adaptive multi-feature layer normalization method, wherein the adaptive multi-feature layer normalization method combines different features and performs layer normalization operation adaptively; In step S3, the parameters of the adaptive multi-feature layer normalization method are dynamically calculated by the full connection layer of the encoding feature map: ； wherein, and respectively represent the encoding of the image obtained by the generator G and the discriminator D, , respectively are and the layer mean of , respectively are and the layer standard deviation of and are parameters generated by a fully connected layer; specifically, and firstly perform an AdaptiveAvgPool operation respectively, splice the obtained features, and then pass them through Two fully connected layers obtain γ and β; In step S1, the residual module of the generator G adds the adaptive multi-feature layer normalization method, and the adaptive multi-feature layer normalization method is added after the up-sampling layer and the convolution layer of the generator G.

2. The dual-encoder-based generative adversarial network image translation method of claim 1, wherein, In step S1, ReLU is used as the activation function in the generator G, and leak-ReLU with a slope of 0.2 is used as the activation function in the discriminator D.

3. The dual-encoder-based generative adversarial network image translation method of claim 1, wherein, In step S1, the loss function expression of the double-encoder generative adversarial network model is ； wherein, represents the adversarial loss function; represents the cycle-consistency loss function; represents the reconstruction loss function , , is a hyper-parameter that controls the weight of each loss function.

4. The dual-encoder-based generative adversarial network image translation method of claim 3, wherein, The expression of the adversarial loss function is ; wherein X is a source domain of images, Y is a target domain of images; x is an image sampled from the source domain, y is an image sampled from the target domain; denotes the result of converting the image x from the source domain into the target domain via the generator G; denotes that the image y is from the target domain Y, denotes that the image x is from the source domain X; denotes the discrimination result of the discriminator on y; denotes the discrimination result of the discriminator on ; denotes the encoding result of the encoder E on ; denotes the encoding part of the discriminator D on the encoding result of denotes the splicing operation.​ 5. The dual-encoder-based generative adversarial network image translation method of claim 3, wherein, The expression of the cycle consistency loss function is ； where x is an image sampled from the source domain; denotes the result of converting the image x from the source domain to the target domain via the generator G; denotes the result of converting the image y from the target domain to the source domain via the generator G; denotes the encoder the encoding result of ; denotes the encoding part of the discriminator D the encoding result of ; denotes the encoding part of the discriminator D the encoding result of ; denotes the concatenation operation; denotes regularization.

6. The dual-encoder-based generative adversarial network image translation method of claim 3, wherein, The expression of the reconstruction loss function is ； wherein, represents that the image x comes from the source domain X; x is an image sampled from the source domain; represents the result of converting the image y from the target domain to the source domain through the generator G; represents the encoding part of the discriminator D to ; represents the splicing operation; represents regular.

7. A computer readable storage medium, the medium storing a computer program, when the computer program is executed by a processor, implementing the method of any one of claims 1-6.

Images