Image style transfer method and device based on cycle consistency

Abstract

The present invention provides an image style transfer method and device based on cycle consistency, and the image style transfer method comprises the following steps: obtaining a photo set to be subjected to image style transfer; based on the photo set, applying a preset encoder in a generator to determine a first deep content feature of a first photo; the first photo is any photo in the photo set; based on the first deep content feature, applying a decoder in the generator to determine a first style transfer image corresponding to the first photo; the image style transfer method and device based on cycle consistency of the present invention can simplify the network architecture and reduce the training overhead; and use the cycle consistency loss as an auxiliary loss to enhance the network style transfer capability, so as to achieve a good style transfer effect, and also allow the user to generate a style transfer image for any photo based on a specified style in real time, thereby enhancing the flexibility and convenience of style transfer.

Description

Technical Field

[0001] The present invention relates to the field of computer applications, and in particular to a method and device for image style transfer based on cycle consistency. Background Art

[0002] Style transfer refers to combining the content of one image with the style of another image to generate a new image. With the maturity of neural network technology, style transfer based on neural networks has gradually become the main method to solve the style transfer problem.

[0003] There are two main types of neural network-based style transfer methods in the prior art, namely deep convolution method and generative adversarial method. The deep convolution method has a fast training speed but poor transfer effect; the generative adversarial method has a strong style transfer capability but a complex network architecture.

[0004] Therefore, how to achieve image style transfer with low complexity and high transfer effect is an urgent problem to be solved. Summary of the invention

[0005] In view of the above-mentioned shortcomings of the prior art, an object of the present invention is to provide a method and device for image style transfer based on cycle consistency, so as to solve the problem that low-complexity and high-transfer effect image style transfer cannot be achieved in the prior art.

[0006] To achieve the above-mentioned purpose and other related purposes, the present invention provides an image style transfer method based on cycle consistency, comprising the following steps: obtaining a photo set to be image style transferred; based on the photo set, applying a preset encoder in a generator to determine a first deep content feature of a first photo; the first photo is any photo in the photo set; based on the first deep content feature, applying a decoder in the generator to determine a first style transfer image corresponding to the first photo; wherein the loss function of the generator includes: a content loss function, a style loss function and a cycle consistency loss function; the content loss function is determined based on a preset training photo set and applied to the generator training; the style loss function is obtained based on the training photo set and the preset style image set, and is applied to the generator training to determine; the cycle consistency loss function is determined based on the training photo set and the style image set, and is applied to the generator training to determine. The content loss function, the style loss function and the cycle consistency loss function are based on a gradient optimization algorithm, and the parameters in the network are optimized by using the values ​​of the loss function obtained in each step of iteration during training to ensure the realization of the style transfer goal.

[0007] In one embodiment of the present invention, the cycle consistency loss function includes an identity loss function and a feature loss function, and the cycle consistency loss function is determined based on the training photo set and the style image set, and is applied to the generator training, including: based on the style image set, applying the encoder to determine the third deep content feature of the first style image; the first style image is any image in the style image set; based on the third deep content feature, applying the decoder to determine the generated image corresponding to the first style image; comparing the first style image and the generated image to determine the identity loss function; based on the training photo set, applying the encoder to determine the second deep content feature of the second photo, the second photo is any photo in the training photo set; based on the second deep content feature, applying the decoder to determine the second style transfer image corresponding to the second deep content feature; based on the second style transfer image, applying the encoder to determine the fourth deep content feature corresponding to the second style transfer image; comparing the second deep content feature with the fourth deep content feature to determine the feature loss function.

[0008] In one embodiment of the present invention, the expression of the identity loss function is:

[0009]

[0010] Among them, L i is the identity loss function; y is the first style image; G(y) is the generated image; i is the order of the style image in the style image set; K is the maximum value of the style images in the style image set.

[0011] In one embodiment of the present invention, the expression of the feature loss function is:

[0012]

[0013] Among them, L f is the feature loss function; f is the second deep content feature; G deo (f) is the second style transfer image; G ene (G dec (f)) is the fourth deep content feature; j is the order of photos in the training photo set; N is the maximum number of photos in the training photo set.

[0014] In one embodiment of the present invention, the expression of the style loss function is:

[0015]

[0016] Among them, L sis the style loss function, x is the second photo; G(x) is the second style transfer image generated corresponding to the second photo; D(G(x)) is the style representation corresponding to the first style transfer image trained by the discriminator; j is the order of the photos in the training photo set; N is the maximum number of photos in the training photo set; 1 represents the true matrix.

[0017] In one embodiment of the present invention, the loss function of the discriminator includes: a false loss function and a true loss function;

[0018] The expression of the false loss function is:

[0019]

[0020] Among them, L df is the false loss function, x is the second photo; G(x) is the second style transfer image generated corresponding to the second photo; D(G(x)) is the style representation corresponding to the second style transfer image; j is the order of the photos in the training photo set; N is the maximum number of photos in the training photo set; 0 represents a false matrix;

[0021] The expression of the true loss function is:

[0022]

[0023] Among them, L dr is the false loss function, y is the first style image; D(y) is the third style transfer image generated corresponding to the first style image; i is the order of the style image in the style image set; K is the maximum value of the style image in the style image set; 1 represents the true matrix.

[0024] In one embodiment of the present invention, the expression of the content loss function is:

[0025]

[0026] Among them, L c is the content loss function; x is the second photo; G(x) is the second style migration image generated corresponding to the second photo; G enc (x) applying a second deep content feature determined by the encoder to the second photo; G enc (G(x)) is the fourth deep content feature determined by applying the encoder to the second style transfer image; j is the order of the photos in the training photo set; N is the maximum number of photos in the training photo set.

[0027] Correspondingly, the present invention provides an image style transfer device based on cycle consistency, comprising: an acquisition module, used to acquire a photo set to be subjected to image style transfer; a first determination module, used to determine a first deep content feature of a first photo based on the photo set and applying a preset encoder in a generator; the first photo is any photo in the photo set; a second determination module, used to determine a first style transfer image corresponding to the first photo based on the first deep content feature and applying a decoder in the generator; wherein the loss function of the generator comprises: a content loss function, a style loss function and a cycle consistency loss function; the content loss function is determined based on a preset training photo set and applied to the generator training; the style loss function is obtained based on the training photo set and a preset style image set and applied to a preset discriminator, and is determined by the generator training; the cycle consistency loss function is determined based on the training photo set and the style image set and applied to the generator training.

[0028] The present invention provides a storage medium on which a computer program is stored. When the program is executed by a processor, the above-mentioned image style transfer method based on cycle consistency is implemented.

[0029] The present invention provides an image style transfer system, comprising a memory for storing a computer program and, according to actual needs, storing an image set generated during actual application; and a processor for running the computer program to implement the above-mentioned image style transfer method based on cycle consistency.

[0030] As described above, the image style transfer method and device based on cycle consistency of the present invention have the following beneficial effects:

[0031] (1) It has strong style transfer capabilities, allowing users to generate style-transferred images for any photo based on a specified style in real time.

[0032] (2) Simplify the network architecture and reduce training costs. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] Figure 1 Shown is a flow chart of an embodiment of a cycle consistency-based image style transfer method of the present invention.

[0034] Figure 2 FIG. 1 is a schematic diagram of a style image set in an embodiment of the image style transfer method based on cycle consistency of the present invention.

[0035] Figure 3 FIG. 4 is a schematic diagram showing a training process of an image style transfer method based on cycle consistency in one embodiment of the present invention.

[0036] Figure 4FIG. 4 is a schematic diagram showing a test flow of an image style transfer method based on cycle consistency in one embodiment of the present invention.

[0037] Figure 5 Shown is a style transfer comparison diagram of an image style transfer method based on cycle consistency in one embodiment of the present invention.

[0038] Figure 6 It is a schematic structural diagram of an image style transfer device based on cycle consistency in one embodiment of the present invention.

[0039] Figure 7 An image style transfer system based on cycle consistency of the present invention is shown in one embodiment.

[0040] Component number description

[0041] 61 Get Module

[0042] 62 First determination module

[0043] 63 Second determination module

[0044] 71 Processor

[0045] 72 Memory DETAILED DESCRIPTION

[0046] The following describes the embodiments of the present invention by specific examples, and those skilled in the art can easily understand other advantages and effects of the present invention from the contents disclosed in this specification. The present invention can also be implemented or applied through other different specific embodiments, and the details in this specification can also be modified or changed in various ways based on different viewpoints and applications without departing from the spirit of the present invention. It should be noted that the following embodiments and features in the embodiments can be combined with each other without conflict.

[0047] It should be noted that the illustrations provided in the following embodiments are only schematic illustrations of the basic concept of the present invention, and thus the drawings only show components related to the present invention rather than being drawn according to the number, shape and size of components in actual implementation. In actual implementation, the type, quantity and proportion of each component may be changed arbitrarily, and the component layout may also be more complicated.

[0048] The image style transfer method and device based on cycle consistency of the present invention simplifies the network architecture and reduces the training overhead by training the basic generative adversarial network; and uses cycle consistency loss as an auxiliary loss to enhance the network style transfer capability to achieve a good style transfer effect. At the same time, it also allows users to generate style transferred images for any photo based on a specified style in real time, thereby enhancing the flexibility and convenience of style transfer.

[0049] like Figure 1 As shown, in this embodiment, the image style transfer method based on cycle consistency of the present invention includes the following steps:

[0050] Step S1: Obtain a photo set to be subjected to image style transfer.

[0051] Specifically, a photo set consisting of real photos to be subjected to image style transfer is obtained.

[0052] Step S2: Based on the photo set, applying a preset encoder in the generator to determine a first deep content feature of a first photo; the first photo is any photo in the photo set.

[0053] Specifically, the generator adopts an encoder-decoder structure, and the generator is first applied to train image style transfer; then the image style transfer is tested.

[0054] Specifically, the basic generative adversarial network architecture is used for training, where the generator is trained using a ResNet-based generator with nine residual blocks, where the encoder includes one input sampling block and two downsampling blocks, and the decoder includes nine residual blocks, two upsampling blocks, and one output sampling block. Each sampling block follows a one-step convolution / deconvolution, normalization, activation layer. Each residual block contains convolution, normalization, activation layer, convolution, normalization, and residual connection, such as Figure 3 As shown, in this embodiment, the training process diagram of the present invention.

[0055] Further specifically, the loss function of the generator includes: a content loss function, a style loss function and a cycle consistency loss function; the content loss function is based on a preset training photo set and is determined by applying the generator training; the style loss function is based on the training photo set and a preset style image set, is obtained by applying a preset discriminator, and is determined by applying the generator training; the cycle consistency loss function is based on the training photo set and the style image set, and is determined by applying the generator training.

[0056] Specifically, the style image set is preprocessed, and the block segmentation method is used to divide the style image into many small image blocks, and these small image blocks are reordered and combined to generate a style image set, so as to expand the scale of the style data set and improve the style transfer capability. Figure 2 As shown, in this embodiment, a schematic diagram of the style image set of the present invention.

[0057] Further specifically, the cycle consistency loss function includes an identity loss function and a feature loss function, and the cycle consistency loss function is determined based on the training photo set and the style image set, applying the generator training, including: based on the style image set, applying the encoder to determine the third deep content feature of the first style image; the first style image is any image in the style image set; based on the third deep content feature, applying the decoder to determine the generated image corresponding to the first style image; comparing the first style image and the generated image to determine the identity loss function; based on the training photo set, applying the encoder to determine the second deep content feature of the second photo, the second photo is any photo in the training photo set; based on the second deep content feature, applying the decoder to determine the second style transfer image corresponding to the second deep content feature; based on the second style transfer image, applying the encoder to determine the fourth deep content feature corresponding to the second style transfer image; comparing the second deep content feature with the fourth deep content feature to determine the feature loss function.

[0058] More specifically, the expression of the identity loss function is:

[0059]

[0060] Among them, L i is the identity loss function; y is the first style image; G(y) is the generated image; i is the order of the style image in the style image set; K is the maximum value of the style images in the style image set.

[0061] More specifically, the expression of the feature loss function is:

[0062]

[0063] Among them, L f is the feature loss function; f is the second deep content feature; G dec (f) is the second style transfer image; G enc (G dec (f)) is the fourth deep content feature; j is the order of photos in the training photo set; N is the maximum number of photos in the training photo set.

[0064] Specifically, the discriminator adopts a 70*70 block structure; the discriminator network contains five down-sampling convolution, normalization, and activation layers, and outputs a discriminated result matrix. Each element of the matrix corresponds to the classification result of a block. This discriminator can process images of any size in a fully convolutional manner.

[0065] The expression of the style loss function is:

[0066]

[0067] Among them, L s is the style loss function, x is the second photo; G(x) is the second style transfer image generated corresponding to the second photo; D(G(x)) is the style representation corresponding to the second style transfer image trained by the discriminator; j is the order of the photos in the training photo set; N is the maximum number of photos in the training photo set; 1 represents the true matrix.

[0068] More specifically, the loss function of the discriminator includes: a false loss function and a true loss function; the expression of the false loss function is:

[0069]

[0070] Among them, L df is the false loss function, x is the second photo; G(x) is the second style transfer image generated corresponding to the second photo; D(G(x)) is the style representation corresponding to the second style transfer image; j is the order of the photos in the training photo set; N is the maximum number of photos in the training photo set; 0 represents a false matrix.

[0071] The expression of the true loss function is:

[0072]

[0073] Among them, L dr is the false loss function, y is the first style image; D(y) is the third style transfer image generated corresponding to the first style image; i is the order of the style image in the style image set; K is the maximum value of the style image in the style image set; 1 represents the true matrix.

[0074] More specifically, the expression of the content loss function is:

[0075]

[0076] Among them, L c is the content loss function; x is the second photo; G(x) is the second style migration image generated corresponding to the second photo; G enc (x) applying a second deep content feature determined by the encoder to the second photo; G enc(G(x)) is the fourth deep content feature determined by applying the encoder to the second style transfer image; j is the order of the photos in the training photo set; N is the maximum number of photos in the training photo set.

[0077] The expression of the total loss function of the generator determined by the above steps is:

[0078] L C =λ c L o +λ s L s +λ i L i +λ f L f

[0079] Among them, L G is the total loss function of the generator, λ c is the weight coefficient of the content loss function; s is the weighted coefficient of the style loss function; i is the weight coefficient of the identity loss function; f is the weight coefficient of the feature loss function.

[0080] The expression of the total loss function of the discriminator determined by the above steps is:

[0081] L D =λ df L df +λ dr L dr

[0082] Among them, L D is the total loss function of the discriminator, λ df is the weight coefficient of the false loss function, λ dr is the weighting coefficient of the true loss function.

[0083] For example, in the image style transfer training process, a general optimizer with β1=0.5, β2=0.999 is used as the gradient descent algorithm, and the training is carried out for a total of eight generations; the learning rate when updating the network parameters is 0.0002, and linear decay begins after half of the total number of iterations; the batch size used here is one, and instance regularization is also used; the network parameters at the beginning of the training are randomly normally distributed. At the same time, in order to enhance the randomness of the photo, the size of the loaded photo is first expanded to 1.12 times the original size, and then randomly cropped to the original size and input into the generator. The size of the loaded photo during training is 256*256 pixels. The size of the style image segmentation is 512*512 pixels. During training, in each iteration, the generator is optimized first, and then the discriminator is optimized. The goal of training is to minimize the total loss function of the generator and the total loss function of the discriminator as much as possible, and based on this, the back-propagation gradient descent algorithm is used to optimize the entire generator and discriminator. The training results show that when the weight coefficients of the generator are λ c =10,λ s =1,λ i =5,λ f = 1, and the weight coefficients of the discriminator are all 0.5, the style transfer effect of the generated image is the best. Specifically, Figure 4 As shown, in this embodiment, the test process diagram of the present invention generates a corresponding style transfer image according to a specified style from a photo input to the generator.

[0084] Step S3: Based on the first deep content feature, apply the decoder in the generator to determine a first style transfer image corresponding to the first photo.

[0085] For example, if Figure 5 As shown in the figure, in this embodiment, the style transfer comparison diagram of the present invention is shown. For different style transfer tasks, the specified style transfer includes: sketch style transfer, ink style transfer and oil painting style transfer. During the test, one of these style transfer pictures is input into the generator network, and the corresponding style transfer image can be generated for the input photo of the generator.

[0086] like Figure 6 As shown, in this embodiment, the image style transfer device based on cycle consistency of the present invention includes:

[0087] An acquisition module 61 is used to acquire a photo set to be subjected to image style transfer;

[0088] A first determination module 62 is used to determine a first deep content feature of a first photo based on the photo set by applying a preset encoder in the generator; the first photo is any photo in the photo set;

[0089] A second determination module 63 is configured to determine a first style transfer image corresponding to the first photo by applying a decoder in the generator based on the first deep content feature;

[0090] Among them, the loss function of the generator includes: a content loss function, a style loss function and a cycle consistency loss function; the content loss function is based on a preset training photo set and is determined by applying the generator training; the style loss function is based on the training photo set and a preset style image set, and is obtained by applying a preset discriminator, and is determined by applying the generator training; the cycle consistency loss function is based on the training photo set and the style image set, and is determined by applying the generator training.

[0091] The technical features specifically implemented in the image style transfer device based on cycle consistency in this embodiment are basically the same as the principles of each step in the image style transfer method based on cycle consistency in Example 1, and the technical contents that are common between the method and the device will not be repeated.

[0092] The storage medium of the present invention stores a computer program, and when the program is executed by a processor, the above-mentioned image style transfer method based on cycle consistency is implemented.

[0093] like Figure 7 As shown, in this embodiment, the electronic device of the present invention includes: a processor 71 and a memory 72 .

[0094] The memory 72 is used to store computer programs.

[0095] The memory 72 includes: ROM, RAM, disk, USB flash drive, memory card or optical disk and other media that can store program codes.

[0096] The processor 71 is connected to the memory 72 and is used to execute the computer program stored in the memory 72 so that the electronic device performs the above-mentioned image style transfer method based on cycle consistency.

[0097] Preferably, the above-mentioned processor 71 can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), a graphic processing unit (GPU), etc., among which, in practical applications, the main general-purpose processor used is a graphics processor; it can also be a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components.

[0098] In summary, the image style transfer method and device based on cycle consistency of the present invention simplifies the network architecture and reduces the training cost by training the basic generative adversarial network; and uses cycle consistency loss as an auxiliary loss to enhance the network style transfer capability to achieve a good style transfer effect, while also allowing users to generate style transfer images for any photo based on a specified style in real time, thereby enhancing the flexibility and convenience of style transfer. Therefore, the present invention effectively overcomes the various shortcomings of the prior art and has a high industrial utilization value.

[0099] The above embodiments are merely illustrative of the principles and effects of the present invention, and are not intended to limit the present invention. Anyone familiar with the art may modify or alter the above embodiments without departing from the spirit and scope of the present invention. Therefore, all equivalent modifications or alterations made by a person of ordinary skill in the art without departing from the spirit and technical concept disclosed by the present invention shall still be covered by the claims of the present invention.

Claims

1. A method for image style transfer based on cycle consistency, characterized in that: The image style transfer method comprises the following steps: Get the photo set to be transferred to image style; Based on the photo set, applying a preset encoder in the generator to determine a first deep content feature of a first photo; the first photo is any photo in the photo set; Based on the first deep content feature, applying a decoder in the generator to determine a first style transfer image corresponding to the first photo; Among them, the loss function of the generator includes: a content loss function, a style loss function and a cycle consistency loss function; the content loss function is based on a preset training photo set and is determined by applying the generator training; the style loss function is based on the training photo set and a preset style image set, and is obtained by applying a preset discriminator, and is determined by applying the generator training; the cycle consistency loss function is based on the training photo set and the style image set, and is determined by applying the generator training; The cycle consistency loss function includes an identity loss function and a feature loss function, and the cycle consistency loss function is determined based on the training photo set and the style image set by applying the generator training, including: Based on the style image set, applying the encoder, determining a third deep content feature of a first style image; the first style image is any image in the style image set; Based on the third deep content feature, applying the decoder, determining a generated image corresponding to the first style image; comparing the first style image and the generated image to determine the identity loss function; Based on the training photo set, applying the encoder, determining a second deep content feature of a second photo, where the second photo is any photo in the training photo set; Based on the second deep content feature, applying the decoder, determining a second style transfer image corresponding to the second deep content feature; Based on the second style transfer image, applying the encoder, determining a fourth deep content feature corresponding to the second style transfer image; The second deep content feature and the fourth deep content feature are compared to determine the feature loss function.

2. The image style transfer method according to claim 1, characterized in that: The expression of the identity loss function is: Among them, L i is the identity loss function; y is the first style image; G(y) is the generated image; i is the order of the style image in the style image set; K is the maximum value of the style images in the style image set.

3. The image style transfer method according to claim 1, characterized in that: The expression of the feature loss function is: Among them, L f is the feature loss function; f is the second deep content feature; G dec (f) is the second style transfer image; G enc (G dec (f)) is the fourth deep content feature; j is the order of photos in the training photo set; N is the maximum number of photos in the training photo set.

4. The image style transfer method according to claim 1, characterized in that: The expression of the style loss function is: Among them, L s is the style loss function, x is the second photo; G(x) is the second style transfer image generated corresponding to the second photo; D(G(x)) is the style representation corresponding to the second style transfer image trained by the discriminator; j is the order of the photos in the training photo set; N is the maximum number of photos in the training photo set; 1 represents the true matrix.

5. The image style transfer method according to claim 4, characterized in that: The loss function of the discriminator includes: a false loss function and a true loss function; The expression of the false loss function is: Among them, L df is the false loss function, x is the second photo; G(x) is the second style transfer image generated corresponding to the second photo; D(G(x)) is the style representation corresponding to the second style transfer image; j is the order of the photos in the training photo set; N is the maximum number of photos in the training photo set; 0 represents a false matrix; The expression of the true loss function is: Among them, L dr is the false loss function, y is the first style image; D(y) is the third style transfer image generated corresponding to the first style image; i is the order of the style image in the style image set; K is the maximum value of the style image in the style image set; 1 represents the true matrix.

6. The image style transfer method according to claim 1, characterized in that: The expression of the content loss function is: Among them, L c is the content loss function; x is the second photo; G(x) is the second style migration image generated corresponding to the second photo; G enc (x) applying a second deep content feature determined by the encoder to the second photo; G enc (G(x)) is the fourth deep content feature determined by applying the encoder to the second style transfer image; j is the order of the photos in the training photo set; N is the maximum number of photos in the training photo set.

7. An image style transfer device based on cycle consistency, characterized in that: include: An acquisition module is used to acquire a photo set to be transferred to an image style; A first determination module, configured to determine a first deep content feature of a first photo by applying a preset encoder in a generator based on the photo set; The first photo is any photo in the photo collection; A second determination module, configured to determine a first style transfer image corresponding to the first photo by applying a decoder in the generator based on the first deep content feature; Among them, the loss function of the generator includes: a content loss function, a style loss function and a cycle consistency loss function; the content loss function is based on a preset training photo set and is determined by applying the generator training; the style loss function is based on the training photo set and a preset style image set, and is obtained by applying a preset discriminator, and is determined by applying the generator training; the cycle consistency loss function is based on the training photo set and the style image set, and is determined by applying the generator training; The cycle consistency loss function includes an identity loss function and a feature loss function, and the cycle consistency loss function is determined based on the training photo set and the style image set by applying the generator training, including: Based on the style image set, applying the encoder, determining a third deep content feature of a first style image; the first style image is any image in the style image set; Based on the third deep content feature, applying the decoder, determining a generated image corresponding to the first style image; comparing the first style image and the generated image to determine the identity loss function; Based on the training photo set, applying the encoder, determining a second deep content feature of a second photo, where the second photo is any photo in the training photo set; Based on the second deep content feature, applying the decoder, determining a second style transfer image corresponding to the second deep content feature; Based on the second style transfer image, applying the encoder, determining a fourth deep content feature corresponding to the second style transfer image; The second deep content feature and the fourth deep content feature are compared to determine the feature loss function.

8. A storage medium storing program instructions, wherein: When the program instructions are executed, the steps of the image style transfer method based on cycle consistency are implemented as described in any one of claims 1 to 6.

9. An image style transfer system, characterized by: The method comprises a memory for storing a computer program; and a processor for running the computer program to implement the steps of the image style transfer method based on cycle consistency as claimed in any one of claims 1 to 6.

Images