A cartoon style transfer method based on Retinex model

Abstract

A cartoon style transfer method based on the Retinex model belongs to the field of computer vision. Converting real-world photos into cartoon-style images is a meaningful and challenging task. Existing methods cannot obtain satisfactory cartoonization results because they do not consider the consistency and continuity of cartoon images and real photos in structure, texture and illumination respectively. In the present invention, the Retinex model jointly learns the intrinsic properties (shape and texture) and extrinsic properties (illumination) of photos and cartoon images. The RexGAN framework includes ReflectGAN, which learns the mapping from photo images to cartoon images through reflection loss, and LuminGAN, which further promotes the structure and illumination of generated images through illumination loss. The present invention can not only generate high-quality cartoon images from real photos, but also has clear edges and structures and correct illumination, and is superior to the most advanced methods in subjective effect.

Description

Technical Field

[0001] The present invention relates to the fields of style transfer, image low-light enhancement, etc., and designs a cartoon style transfer method based on the Retinex decomposition model. The present invention belongs to the field of computer vision, and specifically relates to technologies such as neural style transfer and Retinex image decomposition algorithm. Background Art

[0002] In recent years, science fiction films and TV series have become more and more popular. With the development of computer graphics computing, the production quality of film and television special effects has also made great progress. Producers can achieve more delicate, realistic and shocking picture effects on computers. The current mainstream film and television special effects production software includes Houdini, Nuke and MAYA. The character scenes of the famous animated films "Final Fantasy" and "Madagascar" are all created by MAYA. However, the special effects production process based on these software is complicated, including scene construction, which not only requires a lot of financial and human resources, but also requires professional technicians and a lot of time. Recently, in the field of deep learning, research on style transfer has attracted much attention. Its main goal is to convert ordinary images into paintings with artistic style, which can provide good technical support for the production of film and television special effects. The cartoon art style mainly emphasizes information closely related to the theme, so it simplifies and eliminates the mixed parts. It is precisely because of this unique appeal that this art style is widely used in life, not only in film and television, but also in games, advertising and other fields. Therefore, the research on cartoon style transfer is of great significance.

[0003] Since the 1990s, many non-photorealistic rendering (NPR) algorithms have been developed for specific styles, including cartoon style, oil painting style, and ink painting style. Researchers usually use cel shading or filter processing to obtain specific styles, and these methods are widely used in various software. In recent years, the neural style transfer (NST) method based on convolutional neural network has emerged, which uses the correlation between deep features to achieve good results in painting style. In addition, there is another group of methods based on generative adversarial networks (GANs) to transfer images between two domains in an adversarial manner. Subsequently, a series of methods based on the cycle method were born to complete the domain transfer by training unpaired data. In addition, CartoonGAN, ComixGAN, and AnimeGAN have also achieved great success in cartoon style transfer. However, these methods still show unsatisfactory results in two aspects. 1) The structure and texture of the low-light area of ​​the original image are easily lost; 2) The generated cartoon image does not retain the global color appearance of the original image.

[0004] In the field of low-light enhancement, the Retinex model, as a perceptual model of the human visual system, decomposes the image into illumination and reflection components. Its physical goal can be described as O=I⊙R, that is, decomposing the observed image O into illumination I and reflectivity R, where ⊙ represents the multiplication of elements one by one. In recent years, methods represented by WVM, JeiP, and STAR can decompose images into illumination and reflection components based on Retinex. Among them, the JeiP model can better retain structural information through shape prior, better estimate reflectivity through texture prior, and better obtain light source through illumination prior. Therefore, these characteristics can be used to better convey stylization, improve illumination, and retain the structure of the generated image.

[0005] A generative adversarial network (GAN) framework based on the Retinex model is proposed, denoted as RexGAN. First, by introducing the JeiP-Retinex model, the image is decomposed into its reflectance component to eliminate the influence of external properties (illumination). By utilizing the segmentation of animated images and the piecewise continuous nature of the segmented reflectance component of the image, the mapping of the reflection from the photo image to the cartoon image is learned through an iterative method and reflection loss. Finally, the illumination loss is used to further improve the illumination and protect the structure of the generated image. Summary of the invention

[0006] In the process of realizing image style transfer, the neural style transfer method usually requires that the content image and the style image must have structural similarity, and this method pays more attention to the painting style. Since the generative adversarial network can realize the transfer of two completely different domains, the generative adversarial network has become one of the research hotspots. It has also been applied in style transfer and achieved good results. The most representative ones are CartoonGAN, ComixGAN, and AnimeGAN. However, these methods show unsatisfactory results in two aspects. 1) The structure and texture of the low-light area of ​​the original image are easily lost; 2) The generated cartoon image does not retain the global color appearance of the original image.

[0007] A cartoon style transfer method based on Retinex model, characterized by:

[0008] The implementation of the method is divided into a training phase and a testing phase. First, two generative adversarial networks are trained simultaneously to achieve this, namely the generator G and the discriminator D. R And the generator F and the discriminator D I , then take the real photo x as input and pass it through generator G and generator F continuously to get the output image F(G(x));

[0009] It is divided into three steps: preprocessing of the dataset, training of the RexGAN model, and synthesis of cartoon images;

[0010] 1. Preprocessing of the dataset

[0011] In order to effectively estimate the illumination component I and reflection component R of real photos and cartoon images, the intrinsic and extrinsic joint prior model Jeip is integrated into the mapping function RexGAN; therefore, the Jeip model is first used to decompose the cartoon image y and the real photo x respectively to obtain the reflection component R of the cartoon image y And the reflection component R of the real photo x ; Then use the unpaired training data set and The generator G is trained to learn the mapping function from the photo domain X of the photo image x to the reflection domain of the cartoon image y (ReflectGAN); finally, the paired training dataset and The generator F is trained to learn the mapping function from the reflection domain Q of the cartoon image to the cartoon domain Y (LuminGAN); the training data contains real photos x and cartoon images y, while the test data only includes real photos x; in the training process of RexGAN, the dataset provided by AnimeGAN is used; in addition, all training images are resized to 256*256;

[0012] 2. The training of the RexGAN model is based on the assumption that the reflection component of the image contains fine texture and piecewise continuity, and transfers the reflection of the real photo to the corresponding cartoon image; therefore, the RexGAN model includes two mappings G:x→R y (ReflectGAN) and F:R y →y(LuminGAN); In addition, two adversarial discriminators D are introduced R and D I , D R The purpose is to distinguish the image R y and the transformed image G(x), which corresponds to the generator G; similarly, D I The purpose is to distinguish y from F(R y ), which corresponds to the generator F; therefore, the objective function is expressed as:

[0013]

[0014] Among them, argminmax represents the solution to the maximum and minimum problem. Similar to training a generative adversarial network, it will solve the training problem of two generative adversarial networks, that is, minimizing the generator G and the generator F, and maximizing the discriminator D.R and the discriminator D I ;

[0015] LuminGAN essentially uses a generative adversarial network to preserve the structure of the target image and reconstruct the illumination of the target image; therefore, the formula is derived as:

[0016] L(G, F, D R , D I )=L R (G,D R )+L I (F,D I ) (2)

[0017] L R (G,D R ) and L I (F,D I ) represent the loss functions of ReflectGAN and LuminGAN respectively, which are described in detail below;

[0018] 2.1 Training of ReflectGAN

[0019] ReflectGAN is trained to learn the style features of reflections in cartoon images. In order to reduce the training parameters of ReflectGAN, the generator model in AnimeGANv2 is directly introduced. In addition, a simple block-level discriminator is used to determine whether the generated result has the characteristics of the reflection component of the cartoon image.

[0020] The loss function of ReflectGAN is composed of reflection adversarial loss Content loss Reflective style loss L gra And the color consistency loss L col composition; therefore, L R (G,D R ) is expressed as:

[0021]

[0022] Where ω1=300, ω2=1.4, ω3=2.5 and ω4=100 are weights used to balance the ReflectGAN loss;

[0023] Input a real photo x into the generator G and try to generate an image G(x). The appearance style and texture of the image G(x) should be consistent with the reflective component R of the real cartoon image. y Consistent, while the discriminator D R The purpose is to combine the synthesized image G(x) with the reflection component R yTherefore, the generated image G(x) and the reflected component R are distinguished y Input to the discriminator D R The false probability D R (G(x)) and the true probability D R (R y ), and then the false probability D R (G(x)) is compared with the true probability value 1 and the true probability D R (R y ) is compared with the false probability value 0, thereby making the generator G and the discriminator D R To achieve the purpose of alternating iterative training until convergence; in order to effectively learn the style features of cartoon image reflection, a reflection adversarial loss based on the least squares loss is proposed to constrain the generator G and the discriminator D R ; Reflection countermeasure loss It is expressed as:

[0024]

[0025] in Represents the reflection component R in the reflection domain Q of the cartoon image y The data set, Represents a data set of real photos x in the real photo domain X;

[0026] Perceptual loss is introduced as content loss, which has the ability to maintain image content and overall spatial structure; therefore, VGG’s ability to extract high-level features is used to extract G(x), x, and R y The high-level features of the image are extracted using the Gram matrix to extract the style features reflected by the image from the high-level features of the image. Finally, the content loss and the reflection style loss L gra Defined as:

[0027]

[0028]

[0029] in Represents the reflection component R in the reflection domain Q of the cartoon image y The dataset of real photos x in the real photo domain X and VGG represent the high-level feature maps extracted by the 19-layer VGG network pre-trained on the ImageNet dataset, l represents the feature map of a specific VGG layer; in training, the "conv4-4" layer is selected to calculate this loss;

[0030] The reflection style loss contains the color information of the style image reflection, and the JeiP model mainly performs image decomposition in the illumination (V) channel of the HSV color space; therefore, the image in RGB format is converted to HSV format, and a color consistency loss is established to make the reflection color of the generated image close to the reflection color of the real photo; because the V channel contains a lot of texture information, the l1 sparse constraint is used for the V channel, and the Huber loss l is used for the hue (H) channel and the saturation (S) channel. h ; Color consistency loss L col Defined as:

[0031]

[0032] in Represents the real photo x in the real photo domain X and the real photo reflection component R in the real photo reflection domain P x data set; H(·), S(·), V(·) represent the three channels of the HSV format image, and α represents the weight of the V channel;

[0033] 2.2 LuminGAN Training

[0034] Integrate four lightweight channel attention modules (ECA) into the eight inverse residual blocks (IRBs) of the generator F to form a new residual block;

[0035] Cartoon images and reflections of cartoon images are used as a pair of datasets LuminGAN to train the generator F so that it has the ability to reconstruct lighting features; therefore, the objective function L I (F,D I ) is mainly caused by the loss of light resistance Content loss And the global consistency loss L glo ;The loss function of LuminGAN is expressed as:

[0036]

[0037] Where γ1=150, γ2=0.5 and γ3=1000 are weights used to balance the loss of LuminGAN;

[0038] The reflection component R of the cartoon image y Input into the generator F and try to generate image F(R y ), image F(R y ) should be consistent with the real cartoon image y, while the discriminator D I The purpose is to transform the synthesized image F(R y ) and the reflected component R y to distinguish;

[0039] So the light combats the loss is defined as:

[0040]

[0041] Formula (9) is constrained in the same way as Formula (4), except that the cartoon image y and its reflection component R are respectively y Input to the discriminator D I Get the true probability D I (y) and the false probability D I (F(R y )); Then the false probability D I (F(R y )) is compared with the true probability value 1 and the true probability D I (y) is compared with the false probability value 0, thereby making the generator F and the discriminator D I To achieve the purpose of alternating iterative training until convergence; Represents a data set of cartoon images y in the cartoon image domain Y;

[0042] In order to speed up the convergence of LuminGAN training, a content loss with the same structure as the content loss in ReflectGAN is added. To constrain the generator F, the difference is that only the input real image x in ReflectGAN is changed to the reflection component R of the input cartoon image y ; In order to highlight the edge structure of the image, a color consistency loss constraint generator F in the HSV space is introduced, and this consistency loss is added to the entire image; therefore, the global consistency loss L glo Defined as:

[0043]

[0044] in Represents the reflection component R in the reflection domain Q of the cartoon image y With respect to the data set of cartoon image y in cartoon image domain Y, H(·), S(·), and V(·) respectively represent the three channels of HSV format images, and β=2 represents the weight of the V channel.

[0045] The present invention proposes a generative adversarial network model based on Retinex by jointly considering the intrinsic and extrinsic properties of the image. The model can effectively preserve the color features of the content image and synthesize high-quality cartoon-style images. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] Figure 1 :RexGAN framework diagram

[0047] Figure 2 : Residual module in LuminGAN generator F

[0048] Figure 3 : Comparison of subjective quality with different methods

[0049] Figure 4 : Comparison of three different styles

[0050] Figure 5 : Verify the effect of color consistency loss

[0051] Figure 6 : The impact of different components in the global consistency loss on the results DETAILED DESCRIPTION

[0052] Figure 1 As shown in Figure 1, the implementation of the method is divided into a training phase and a testing phase. The method first trains two generative adversarial networks at the same time, namely the generator G and the discriminator D. R And the generator F and the discriminator D I , then take the real photo x as input and make it pass through generator G and generator F continuously to finally get the output image F(G(x)). The method can be divided into the following three steps: preprocessing of the dataset, training of the RexGAN model, and synthesis of cartoon images.

[0053] 3. Preprocessing of the dataset

[0054] In order to effectively estimate the illumination component I and reflection component R of real photos and cartoon images, the intrinsic and extrinsic joint prior model (Je ip) is integrated into the mapping function (RexGAN). Therefore, the Jeip model is first used to decompose the cartoon image y and the real photo x respectively to obtain the reflection component R of the cartoon image. y And the reflection component R of the real photo x Then, using the unpaired training dataset and The generator G is trained to learn the mapping function from the reflection domain P of the photo image x to the reflection domain Q of the cartoon image y (ReflectGAN). Finally, through the paired training dataset and The generator F is trained to learn the mapping function from the reflection domain Q of cartoon images to the cartoon domain Y (LuminGAN). The training data contains real photos x and cartoon images y, while the test data only includes real photos x. In the training process of RexGAN, the dataset provided by AnimeGAN is used. This dataset contains 6656 real photos as content image datasets, and key frames cut from three cartoon movies (directed by Hayao Miyazaki, Makoto Shinkai and Satoshi Kon respectively) as style image datasets, where different authors represent different styles. In addition, all training images are resized to 256*256.

[0055] 4. The training of the RexGAN model is based on the assumption that the reflection component of the image contains fine texture and piecewise continuity, and transfers the reflection of the real photo to the corresponding cartoon image. Therefore, the RexGAN model includes two mappings G: x→R y (ReflectGAN) and F:R y →y(LuminGAN). In addition, two adversarial discriminators D are introduced R and D I , D R The purpose is to distinguish the image R y and the transformed image G(x), which corresponds to the generator G; similarly, D I The purpose is to distinguish y from F(R y ), which corresponds to the generator F. Therefore, the objective function is expressed as:

[0056]

[0057] Among them, argminmax represents the solution to the maximum and minimum problem. Similar to training a generative adversarial network, it will solve the training problem of two generative adversarial networks, that is, minimizing the generator G and the generator F, and maximizing the discriminator D. R and the discriminator D I .

[0058] Since the appearance of an object is affected by its intrinsic and extrinsic properties, the intrinsic properties of an object, including shape and texture, are independent of lighting. On this basis, the reflection layer is used to establish the relationship between the reflection layer of a real photo and the reflection layer of a cartoon image, and to learn the style features of the cartoon image. LuminGAN essentially uses a generative adversarial network to preserve the structure of the target image and reconstruct the lighting of the target image. Therefore, the formula can be derived as:

[0059] L(G, F, D R , D I )=L R (G,D R )+L I (F,DI ) (2)

[0060] L R (G,D R ) and L I (F,D I ) represent the loss functions of ReflectGAN and LuminGAN respectively, which are described in detail below.

[0061] 4.1 Training of ReflectGAN

[0062] ReflectGAN is trained to learn the style features of reflections in cartoon images. In order to reduce the training parameters of ReflectGAN, the generator model in AnimeGANv2 is directly introduced. In addition, a simple block-level discriminator is used to determine whether the generated result has the characteristics of the reflection component of the cartoon image.

[0063] The loss function of ReflectGAN is composed of reflection adversarial loss Content loss Reflective style loss L qra And the color consistency loss L col Therefore, L R (G,D R ) can be expressed as:

[0064]

[0065] Among them, ω1=300, ω2=1.4, ω3=2.5 and ω4=100 are weights used to balance the loss of ReflectGAN. The weights in this section are obtained by a large number of experiments.

[0066] Input a real photo x into the generator G and try to generate an image G(x). The appearance style and texture of the image G(x) should be consistent with the reflective component R of the real cartoon image. y Consistent, while the discriminator D R The purpose is to combine the synthesized image G(x) with the reflection component R y Therefore, the generated image G(x) and the reflected component R y Input to the discriminator D R The false probability D R (G(x)) and the true probability D R (R y ), and then the false probability D R (G(x)) is compared with the true probability value 1 and the true probability D R (R y ) is compared with the false probability value 0, thereby making the generator G and the discriminator DR To achieve the purpose of alternating iterative training until convergence. In order to effectively learn the style features of cartoon image reflection, a reflection adversarial loss based on the least squares loss is proposed to constrain the generator G and the discriminator D R . Reflection countermeasure loss It can be expressed as:

[0067]

[0068] in Represents the reflection component R in the reflection domain Q of the cartoon image y The data set, Represents a data set of real photos x in the real photo domain X.

[0069] The perceptual loss is introduced as the content loss, which has the ability to maintain the image content and overall spatial structure. Therefore, VGG’s ability to extract high-level features is used to extract G(x), x, and R y In addition, the Gram matrix is ​​used to extract the style features reflected by the image from the high-level features of the image. Finally, the content loss and the reflection style loss L gra Defined as:

[0070]

[0071]

[0072] in Represents the reflection component R in the reflection domain Q of the cartoon image y The dataset of real-life photos x in the real-life photo domain X and VGG denotes the high-level feature maps extracted by a 19-layer VGG network pre-trained on the ImageNet dataset, and l denotes the feature map of a specific VGG layer. In training, the “conv4-4” layer is selected to calculate this loss.

[0073] The reflection style loss contains the color information of the style image reflection, while the JeiP model mainly performs image decomposition in the illumination (V) channel of the HSV color space. Therefore, the image in RGB format is converted to HSV format, and a color consistency loss is established to make the reflection color of the generated image close to the reflection color of the real photo. Because the V channel contains a lot of texture information, the l1 sparse constraint is used for the V channel, and the Huber loss l2 is used for the hue (H) channel and the saturation (S) channel. h . Color consistency loss L col Defined as:

[0074]

[0075] in Represents the real photo x in the real photo domain X and the real photo reflection component R in the real photo reflection domain P x H(·), S(·), V(·) represent the three channels of the HSV format image, and α represents the weight of the V channel.

[0076] 4.2 Training of LuminGAN

[0077] The purpose of LuminGAN is to reconstruct the lighting components of cartoon images. Therefore, the generator of AnimeGAN is directly used together with the traditional pixel-level discriminator. In order to avoid high-frequency artifacts in the generated images and reduce the complexity of training, such as Figure 2 As shown, four lightweight channel attention modules (ECA) are integrated into the eight inverse residual blocks (IRBs) of the generator F to form a new residual block.

[0078] In order to make the generated images have clearer edges and better visual perception, cartoon images and reflections of cartoon images are used as a pair of datasets LuminGAN to train the generator F so that it has the ability to reconstruct lighting features. Therefore, the objective function L I (F,D I ) is mainly caused by the loss of light resistance Content loss And the global consistency loss L glo The loss function of LuminGAN is expressed as:

[0079]

[0080] Where γ1=150, γ2=0.5 and γ3=1000 are weights used to balance the loss of LuminGAN.

[0081] The reflection component R of the cartoon image y Input into the generator F and try to generate image F(R y ), image F(R y ) should be consistent with the real cartoon image y, while the discriminator D I The purpose is to transform the synthesized image F(R y ) and the reflected component R y Distinguish.

[0082] So the light combats the loss is defined as:

[0083]

[0084] Formula (9) is constrained in the same way as Formula (4), except that the cartoon image y and its reflection component R are respectivelyy Input to the discriminator D I Get the true probability D I (y) and the false probability D I (F(R y )). Then the false probability D I (F(R y )) is compared with the true probability value 1 and the true probability D I (y) is compared with the false probability value 0, thereby making the generator F and the discriminator D I To achieve the purpose of alternating iterative training until convergence. Represents a data set of cartoon images y in the cartoon image domain Y.

[0085] In order to speed up the convergence of LuminGAN training, a content loss with the same structure as the content loss in ReflectGAN is added. To constrain the generator F, the difference is that only the input real image x in ReflectGAN is changed to the reflection component R of the input cartoon image y In order to highlight the edge structure of the image, a color consistency loss constraint generator F in the HSV space is introduced, and this consistency loss is added to the entire image. Therefore, the global consistency loss L glo Defined as:

[0086]

[0087] in Represents the reflection component R in the reflection domain Q of the cartoon image y With respect to the data set of cartoon image y in cartoon image domain Y, H(·), S(·), and V(·) respectively represent the three channels of HSV format images, and β=2 represents the weight of the V channel.

[0088] 5. Cartoon image synthesis

[0089] The generator G first converts the input photo image x into G(x) whose statistical features are similar to the reflection component of the cartoon image, and then the generator F converts G(x) into a cartoon image F(G(x)).

[0090] In order to verify the effectiveness of the solution proposed in this invention, experiments were conducted from different aspects. The results were compared with the four most advanced works. Figure 3 The result not only retains the texture information of the real photo, but also reproduces the clear edges, and the overall color is more in line with the visual effect. In addition, we try to test the method from different styles. The results are as follows Figure 4 shown.

[0091] Figure 5It shows how color consistency loss affects the generation results of ReflectGAN. Content loss and reflection style loss enable generator G to generate stylized images, but they are prone to over-stylization. After using color consistency loss, images (c) and (d) have clear textures. However, different α has a significant impact on the results. Compared with the results of α=1, the results of α=2 show more correct colors. In order to further study the impact of different compositions of global consistency loss on LuminGAN, β=2 is set in formula (11), and the different effects of the three compositions of this loss on the results are compared. As Figure 6 As shown in (d), although the global consistency of the RGB format presents a good appearance, its structure is blurred. Similarly, the results presented by using only the HSV format as a constraint are opposite to those of the RGB format. In contrast to them, image (d) shows clear edges and a satisfactory color appearance.

Claims

1. A cartoon style transfer method based on Retinex model, characterized by: The implementation of the method is divided into a training phase and a testing phase. First, two generative adversarial networks are trained simultaneously to achieve this, namely the generator G and the discriminator D. R And the generator F and the discriminator D I , then take the real photo x as input and make it pass through generator G and generator F continuously to finally get the output image F(G(x)); It is divided into three steps: preprocessing of the dataset, training of the RexGAN model, and synthesis of cartoon images; 1) Preprocessing of the dataset In order to effectively estimate the illumination component I and reflection component R of real photos and cartoon images, the intrinsic and extrinsic joint prior model Jeip is integrated into the mapping function RexGAN; therefore, the Jeip model is first used to decompose the cartoon image y and the real photo x respectively to obtain the reflection component R of the cartoon image y And the reflection component R of the real photo x ; Then use the unpaired training data set and The generator G is trained to learn the mapping function ReflectGAN from the photo domain X of the photo image x to the reflection domain Q of the cartoon image y; finally, the paired training dataset and The generator F is trained to learn the mapping function LuminGAN from the reflection domain Q of the cartoon image to the cartoon domain Y; the training data contains real photos x and cartoon images y, while the test data only includes real photos x; in the training process of RexGAN, the dataset provided by AnimeGAN is used; in addition, all training images are resized to 256*256; 2) Training of the RexGAN model Based on the assumption that the reflection component of the image contains fine texture and piecewise continuity, the reflection of the real photo is transferred to the corresponding cartoon image; therefore, the RexGAN model includes two mappings G:x→R y That is, ReflectGAN and F:R y →y is LuminGAN; in addition, two adversarial discriminators D are introduced R and D I , D R The purpose is to distinguish the image R y and the transformed image G(x), which corresponds to the generator G; similarly, D I The purpose is to distinguish y from F(R y ), which corresponds to the generator F; therefore, the objective function is expressed as: Among them, argminmax represents the solution to the maximum and minimum problem. Similar to training a generative adversarial network, it will solve the training problem of two generative adversarial networks, that is, minimizing the generator G and the generator F, and maximizing the discriminator D. R and the discriminator D I ; LuminGAN essentially uses a generative adversarial network to preserve the structure of the target image and reconstruct the illumination of the target image; therefore, the formula is derived as: L(G,F,D R ,D I )=L R (G,D R )+L I (F,D I ) (2) L R (G,D R ) and L I (F,D I ) represent the loss functions of ReflectGAN and LuminGAN respectively, which are described in detail below; 2.1 Training of ReflectGAN ReflectGAN is trained to learn the style features of reflections in cartoon images. In order to reduce the training parameters of ReflectGAN, the generator model in AnimeGANv2 is directly introduced. In addition, a simple block-level discriminator is used to determine whether the generated result has the characteristics of the reflection component of the cartoon image. The loss function of ReflectGAN is composed of reflection adversarial loss Content loss Reflective style loss L gra And the color consistency loss L col composition; therefore, L R (G,D R ) is expressed as: Where ω1=300, ω2=1.4, ω3=2.5 and ω4=100 are weights used to balance the loss of ReflectGAN; Input a real photo x into the generator G and try to generate an image G(x). The appearance style and texture of the image G(x) should be consistent with the reflective component R of the real cartoon image. y Consistent, while the discriminator D R The purpose is to combine the synthesized image G(x) with the reflection component R y Therefore, the generated image G(x) and the reflected component R are distinguished y Input to the discriminator D R The false probability D R (G(x)) and the true probability D R (R y ), and then the false probability D R (G(x)) is compared with the true probability value 1 and the true probability D R (R y ) is compared with the false probability value 0, thereby making the generator G and the discriminator D R To achieve the purpose of alternating iterative training until convergence; in order to effectively learn the style features of cartoon image reflection, a reflection adversarial loss based on the least squares loss is proposed to constrain the generator G and the discriminator D R ; Reflection countermeasure loss It is expressed as: in Represents the reflection component R in the reflection domain Q of the cartoon image y data set, Represents a data set of real photos x in the real photo domain X; Perceptual loss is introduced as content loss, which has the ability to maintain image content and overall spatial structure; therefore, VGG’s ability to extract high-level features is used to extract G(x), x, and R y The high-level features of the image are extracted using the Gram matrix to extract the style features reflected by the image from the high-level features of the image. Finally, the content loss and the reflection style loss L gra Defined as: in Represents the reflection component R in the reflection domain Q of the cartoon image y The dataset of real photos x in the real photo domain X and VGG represent the high-level feature maps extracted by the 19-layer VGG network pre-trained on the ImageNet dataset, l represents the feature map of a specific VGG layer; in training, the "conv4-4" layer is selected to calculate this loss; The reflection style loss contains the color information of the style image reflection, and the JeiP model mainly performs image decomposition in the illumination (V) channel of the HSV color space; therefore, the image in RGB format is converted to HSV format, and a color consistency loss is established to make the reflection color of the generated image close to the reflection color of the real photo; because the V channel contains a lot of texture information, the l1 sparse constraint is used for the V channel, and the Huber loss l is used for the hue (H) channel and the saturation (S) channel. h ; Color consistency loss L col Defined as: in Represents the real photo x in the real photo domain X and the real photo reflection component R in the real photo reflection domain P x data set; H(·), S(·), V(·) represent the three channels of HSV format image, and α represents the weight of V channel; 2.2 LuminGAN Training Integrate four lightweight channel attention modules ECA into the eight inverse residual blocks IRB of the generator F to form a new residual block; Cartoon images and reflections of cartoon images are used as a pair of datasets LuminGAN to train the generator F so that it has the ability to reconstruct lighting features; therefore, the objective function L I (F,D I ) is mainly caused by the loss of light resistance Content loss And the global consistency loss L glo ;The loss function of LuminGAN is expressed as: Where γ1=150, γ2=0.5 and γ3=1000 are weights used to balance the loss of LuminGAN; The reflection component R of the cartoon image y Input to the generator F and try to generate image F(R y ), image F(R y ) should be consistent with the real cartoon image y, while the discriminator D I The purpose is to transform the synthesized image F(R y ) and the reflected component R y to distinguish; So the light combats the loss is defined as: Formula (9) is constrained in the same way as Formula (4), except that the cartoon image y and its reflection component R are respectively y Input to the discriminator D I Get the true probability D I (y) and the false probability E I (F(R y )); Then the false probability D I (F(R y )) is compared with the true probability value 1 and the true probability D I (y) is compared with the false probability value 0, thereby making the generator F and the discriminator D I To achieve the purpose of alternating iterative training until convergence; Represents a data set of cartoon images y in the cartoon image domain Y; In order to speed up the convergence of LuminGAN training, a content loss with the same structure as the content loss in ReflectGAN is added. To constrain the generator F, the difference is that only the input real image x in ReflectGAN is changed to the reflection component R of the input cartoon image y ; In order to highlight the edge structure of the image, a color consistency loss constraint generator F in the HSV space is introduced, and this consistency loss is added to the entire image; therefore, the global consistency loss L glo Defined as: in Represents the reflection component R in the reflection domain Q of the cartoon image y With respect to the data set of cartoon image y in cartoon image domain Y, H(·), S(·), and V(·) respectively represent the three channels of the HSV format image, and β=2 represents the weight of the V channel.

Images