A human style transfer method, device and medium based on generative adversarial network

Abstract

The present invention relates to the field of image analysis and specifically discloses a method, device, and medium for human style transfer based on a generative adversarial network. The method comprises identifying the region of a transfer target in a source image and outputting the region as a first segmentation target; identifying the region of a transfer target in a reference image and outputting the region as a second segmentation target; mapping the first segmentation target onto a first intermediate domain to obtain a third segmentation target; performing style transfer on the third segmentation target based on the second segmentation target to generate a first map; overlaying the first map onto the region of the first segmentation target in the source image, and outputting the covered source image as a generated image. By segmenting the transfer target, the present invention ensures the accuracy of the style transfer object; and by setting an intermediate domain, it overcomes the problem that different source images may each obey multiple different distributions.

Description

Technical Field

[0001] The present invention relates to the field of image analysis, and in particular to a method, device and medium for human style transfer based on a generative adversarial network. Background Art

[0002] With the rise and development of mobile internet, short videos, live streaming, and image beautification applications have become increasingly ubiquitous in our lives. Human style transfer, as a special effects technology, is a crucial component of these applications. This technology can achieve operations such as transferring skin color and hair color. Furthermore, in e-commerce scenarios, it can be used to transfer the style of models, allowing users to understand products from multiple angles while reducing the number of model shots, saving time and costs. Therefore, research on human style transfer technology is of great value.

[0003] In recent years, with the advancement of deep learning and computer vision, deep convolutional neural networks have been widely used in style transfer research. This research can be broadly divided into two categories. One is based on direct mapping, exemplified by pixel2pixel. This model consists of a generator and a discriminator. The generator is responsible for converting an image from a source style (source domain) to a target style (target domain), while the discriminator determines whether the generated image has similar features to the corresponding target domain image. This approach directly establishes a mapping model from the source domain to the target domain. The other is based on a reference image, exemplified by PSGAN. This approach requires an arbitrary image with target domain features as a reference image. When performing the transformation using the generator, the features of the reference image are incorporated, thereby achieving transfer from the source domain to the target domain. Compared to direct mapping models, this approach aims to model the transfer rules, rather than being limited to a specific target domain.

[0004] While both of these methods can accomplish the task of human style transfer to a certain extent, they both present some significant challenges. Pixel-to-pixel methods require paired data for effective training, meaning a sample must have both a source domain image and a corresponding target domain image for training, a requirement that is difficult to achieve with conventional data. PSGAN methods, due to a lack of strict data constraints, may result in multiple different distributions for source and reference images, leading to suboptimal model performance. Furthermore, this method uses the entire image as input, which can cause changes in some areas unrelated to the transfer. Summary of the Invention

[0005] In order to overcome the problems that the existing style transfer process may affect areas other than the transfer target, and that different source domain images may obey multiple different distributions, the present invention provides a human body style transfer method, device and medium based on a generative adversarial network.

[0006] The technical solution adopted by the present invention is: a human style transfer method based on a generative adversarial network, comprising:

[0007] Identify the region where the migration target is located in the source domain image and output the region as the first segmentation target;

[0008] Identify the region where the migration target is located in the reference image, and output the region as the second segmentation target;

[0009] Mapping the first segmentation target onto the first intermediate domain to obtain a third segmentation target;

[0010] Performing style transfer on the third segmentation target based on the second segmentation target to generate a first map;

[0011] The first map is covered to the area of the first segmentation target in the source domain image, and the covered source domain image is output as a generated image.

[0012] Preferably, before outputting the covered source domain image as the generated image, the method further includes:

[0013] Verifying the style transfer effect of the generated image relative to the source domain image and the reference image through a total loss function; wherein the total loss function includes an adversarial loss function, a gradient difference loss function, a cycle loss function, and a color loss function;

[0014] The adversarial loss function is used to verify the distribution difference between the first map and the second segmentation target;

[0015] The gradient difference loss function is used to prioritize the structural information of the migration target;

[0016] The loop loss function is used to prioritize the global consistency information of the migration target;

[0017] The color loss function is used to prioritize maintaining the color information of the migration target.

[0018] Preferably, when the source domain image and the reference image are paired, the total loss function further includes a minimum square error loss function; the minimum square error loss function is used to calculate the L2 loss between the generated image and the second segmentation target.

[0019] Preferably, the steps of identifying the region where the migration target is located in the source domain image and outputting the region as the first segmentation target and identifying the region where the migration target is located in the reference image and outputting the region as the second segmentation target are as follows:

[0020] Establish a segmentation model based on the BASNet saliency detection algorithm;

[0021] Collect images containing the migration target to establish a first training set and a first test set;

[0022] Training and testing the segmentation model using the first training set and the first test set;

[0023] Inputting the source domain image into a segmentation model and outputting a first segmentation target;

[0024] The reference image is input into a segmentation model, and a second segmentation target is output.

[0025] Preferably, mapping the first segmentation target onto the first intermediate domain to obtain the third segmentation target is specifically:

[0026] Based on the pixel2pixel model architecture, an intermediary domain conversion model is established;

[0027] Collect images containing the first segmented target to establish a second training set and a second test set;

[0028] Training and testing the intermediate domain conversion model using the second training set and the second test set;

[0029] The first segmentation target is mapped to the first intermediate domain through an intermediate domain conversion model to obtain a third segmentation target.

[0030] The present invention also provides a human style transfer device based on a generative adversarial network, comprising: a first segmentation module, a second segmentation module, an intermediate domain conversion module, a transfer module, a generation module, and a verification module;

[0031] The first segmentation module is used to identify the region where the migration target is located in the source domain image, and output the region as a first segmentation target;

[0032] The second segmentation module is used to identify the area where the migration target is located in the reference image, and output the area as a second segmentation target;

[0033] The intermediate domain conversion module is used to map the first segmentation target to the first intermediate domain to obtain a third segmentation target;

[0034] The migration module is used to perform style migration on the third segmentation target based on the second segmentation target to generate a first map;

[0035] The generating module is configured to cover the first map to the area of the first segmentation target in the source domain image, and output the covered source domain image as a generated image.

[0036] Preferably, it also includes a verification module;

[0037] The verification module is used to verify the style transfer effect of the generated image relative to the source domain image and the reference image through the total loss function;

[0038] The verification module includes an adversarial loss unit, a gradient difference loss unit, a cycle loss unit and a color loss unit;

[0039] The adversarial loss unit is used to verify the distribution difference between the first map and the second segmentation target;

[0040] The gradient difference loss unit is used to prioritize the structural information of the migration target;

[0041] The cycle loss unit is used to prioritize maintaining global consistency information of the migration target;

[0042] The color loss unit is used to preferentially maintain color information of the migration target.

[0043] Preferably, the verification module further includes a first judgment unit and a minimum square error loss unit;

[0044] The first judging unit is used to judge whether the source domain image and the reference image are paired;

[0045] The least square error loss unit is used to calculate the L2 loss between the generated image and the second segmentation target when the first judgment unit determines that the result is yes.

[0046] The present invention also provides a computer-readable storage medium, which includes a stored computer program, wherein when the computer program is running, the device where the computer-readable storage medium is located is controlled to execute the above-mentioned human style transfer method based on generative adversarial network.

[0047] The beneficial effects of the present invention are:

[0048] (1) Before style transfer, the target area in the image is segmented and style transfer is performed on it separately, while the rest of the image remains unchanged, thus ensuring the accuracy of style transfer.

[0049] (2) By setting an intermediate domain, the source domains are first mapped to the same intermediate domain before style transfer, thus overcoming the problem that different source domain images may obey multiple different distributions;

[0050] Preferably, there is no need to impose many restrictions on the image data used for training and migration. Both paired and unpaired images can be used for normal training and migration, which greatly reduces the difficulty and cost of collecting image data for style transfer model training. BRIEF DESCRIPTION OF THE DRAWINGS

[0051] The present invention will be further described below with reference to the accompanying drawings, in which:

[0052] Figure 1 A flow chart of a method according to one embodiment of the present invention;

[0053] Figure 2 is an overall architecture diagram of another embodiment of the present invention;

[0054] Figure 3 This is a schematic diagram of a segmentation module according to another embodiment of the present invention.

[0055] In the figure: 1. Source domain image; 2. Reference image; 3. First segmentation target; 4. Second segmentation target; 5. First intermediate domain; 6. Third segmentation target; 7. First segmentation module; 8. Second segmentation module; 10. First map; 11. Reconstructed image; 12. Generated image. DETAILED DESCRIPTION

[0056] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. All other embodiments obtained by ordinary technicians in this field based on the embodiments of the present invention without making any creative efforts shall fall within the scope of protection of the present invention.

[0057] See also Figures 1 to 3 As one of the embodiments of the present invention, a method, device and medium for human style transfer based on a generative adversarial network are specifically disclosed.

[0058] The human body style transfer method of this embodiment includes the following steps:

[0059] S1, identifying the region where the migration target is located in the source domain image 1, and outputting the region as the first segmentation target 3;

[0060] S2, identifying the region where the migration target is located in the reference image 2, and outputting the region as the second segmentation target 4;

[0061] S3, mapping the first segmentation target 3 to the first intermediate domain 5 to obtain a third segmentation target 6;

[0062] S4, performing style transfer on the third segmentation target 6 according to the second segmentation target 4 to generate a first map 10;

[0063] S5 , covering the first map 10 to the area of the first segmentation target 3 in the source domain image 1 , and outputting the covered source domain image 1 as a generated image 12 .

[0064] Preferably, before outputting the covered source domain image 1 as the generated image 12 in step S5, the following sub-steps are further included:

[0065] S51. Verify the style transfer effect of the generated image 12 relative to the source domain image 1 and the reference image 2 through the total loss function.

[0066] The total loss function includes an adversarial loss function, a gradient difference loss function, a cycle loss function, and a color loss function.

[0067] The adversarial loss function is used to verify the distribution difference between the first map 10 and the second segmentation target 4;

[0068] The gradient difference loss function is used to prioritize the structural information of the migration target;

[0069] The loop loss function is used to prioritize the global consistency information of the migration target;

[0070] The color loss function is used to prioritize maintaining the color information of the migration target.

[0071] Preferably, the step S5 further includes the following sub-steps:

[0072] S52: When the source domain image 1 and the reference image 2 are paired, the total loss function further includes a minimum square error loss function.

[0073] The least square error loss function is used to calculate the L2 loss between the generated image 12 and the second segmentation target 4.

[0074] Preferably, in step S1 and step S2, the steps of identifying the region where the migration target is located in the source domain image 1 and outputting the region as the first segmentation target 3, and identifying the region where the migration target is located in the reference image 2 and outputting the region as the second segmentation target 4 are specifically implemented as follows:

[0075] A1. Build a segmentation model based on the BASNet saliency detection algorithm.

[0076] A2. Collect images containing the migration target and establish a first training set and a first test set;

[0077] A3. Training and testing the segmentation model using the first training set and the first test set;

[0078] A4. Input the source domain image 1 into the segmentation model and output a first segmentation target 3;

[0079] A5. Input the reference image 2 into the segmentation model and output the second segmentation target 4.

[0080] Preferably, in step S3, the first segmentation target 3 is mapped to the first intermediate domain 5 to obtain a third segmentation target 6, and the specific implementation steps are as follows:

[0081] B1. Based on the pixel2pixel model architecture, establish an intermediary domain conversion model;

[0082] B2. Collect images containing the first segmented target 3 to establish a second training set and a second test set;

[0083] B3. Training and testing the intermediate domain conversion model using the second training set and the second test set;

[0084] B4. Map the first segmentation target 3 to the first intermediate domain 5 through the intermediate domain conversion model to obtain a third segmentation target 6.

[0085] The present invention also provides a human style transfer device based on a generative adversarial network, comprising: a first segmentation module 7, a second segmentation module 8, an intermediate domain conversion module, a transfer module, a generation module and a verification module;

[0086] The first segmentation module 7 is used to identify the region where the migration target is located in the source domain image 1, and output the region as the first segmentation target 3;

[0087] The second segmentation module 8 is used to identify the area where the migration target is located in the reference image 2, and output the area as the second segmentation target 4;

[0088] The intermediate domain conversion module is used to map the first segmentation target 3 to the first intermediate domain 5 to obtain a third segmentation target 6;

[0089] The migration module is used to perform style migration on the third segmentation target 6 based on the second segmentation target 4 to generate a first map 10;

[0090] The generating module is used to cover the first map 10 to the area of the first segmentation target 3 in the source domain image 1, and output the covered source domain image 1 as the generated image 12.

[0091] Preferably, it also includes a verification module;

[0092] The verification module is used to verify the style migration effect of the generated image 12 relative to the source domain image 1 and the reference image 2 through the total loss function;

[0093] The verification module includes an adversarial loss unit, a gradient difference loss unit, a cycle loss unit and a color loss unit;

[0094] The adversarial loss unit is used to verify the distribution difference between the first map 10 and the second segmentation target 4;

[0095] The gradient difference loss unit is used to prioritize the structural information of the migration target;

[0096] The cycle loss unit is used to prioritize maintaining global consistency information of the migration target;

[0097] The color loss unit is used to preferentially maintain color information of the migration target.

[0098] Preferably, the verification module further includes a first judgment unit and a minimum square error loss unit;

[0099] The first judging unit is used to judge whether the source domain image 1 and the reference image 2 are paired;

[0100] The least square error loss unit is used to calculate the L2 loss between the generated image 12 and the second segmentation target 4 when the first judgment unit determines that the result is yes.

[0101] As another embodiment of the present invention, the above embodiment is applied to a human hair color transfer scenario, which is further described in detail. The transfer target in this embodiment is human hair, and the framework for hair color transfer is similar to that for skin color transfer. The model program in this embodiment includes four modules: a hair segmentation module, an intermediate domain conversion module, a generation module, and a supervision module.

[0102] This embodiment preprocesses data using a segmentation module and an intermediate domain conversion module. These two modules are responsible for segmenting the hair area from the image as the segmentation target and converting the segmentation target to a common intermediate domain. In this embodiment, the color of the segmented hair is converted to brown. Because these two modules are preprocessing modules, the parameters of the models involved are all determined in advance.

[0103] After the image completes the intermediate domain conversion, it will enter the generation module to complete the hair color style transfer and obtain the hair map after style transfer; finally, the cycle consistency loss, gradient difference loss, color loss, adversarial loss and supervision module are combined as the total loss function of the model. After obtaining the final generated image, the first segmentation target 3 obtained by the segmentation module is combined, and the first map 10 after style transfer is pasted back to the source domain image 1 to obtain the generated image 12.

[0104] The specific implementation method of this embodiment is as follows:

[0105] The hair segmentation module is used to identify the area occupied by hair in the image and correctly segment it for subsequent transformation operations. The hair segmentation module of this embodiment uses the BASNet saliency detection algorithm and is mainly composed of two parts: the prediction unit (PM) and the residual enhancement unit (RRM).

[0106] The prediction unit is used to roughly segment the image. Its structure is a classic encoder-decoder network. The encoder is responsible for extracting image features and consists of an input convolutional layer followed by six convolutional stages. The input layer and the first four convolutional stages are similar to the structure of ResNet-34, except that the input layer uses 64 3x3 convolution kernels with a stride of 1 and no pooling operation. The last two convolutional stages each consist of three basic res-blocks, which are convolved with 512 convolution kernels after a non-overlapping max pooling of size 2. Through the operations of the encoder, high-level semantic features with gradually decreasing resolution are ultimately obtained. The decoder is responsible for gradually restoring and amplifying the high-level semantic information, thereby gradually obtaining high-resolution feature maps. Its structure is symmetrical to the encoder, except that the downsampling operation is replaced by an upsampling operation. Finally, in order to make the final output feature map compatible with high-level features and low-level features, there will be a bridge between the corresponding Encoder and Decoder. The bridge consists of three convolutional layers, each of which has 512 3x3 convolution kernels with a dilation factor of 2, and each layer has Batch Normalization and ReLU activation functions at the end.

[0107] The residual enhancement unit is responsible for refining the coarse segmentation. Its structure is similar to that of the prediction unit, except that the network layers are shallower, and the encoder has only four convolution stages. Each convolution stage has a consistent structure, consisting of 64 3x3 convolution kernels, batch normalization, and ReLU activation functions. The decoder and encoder are also symmetrical. To ensure global and local refinement of the output, the output of the residual enhancement unit is cascaded with the output of the prediction unit, and a bridge is also provided between the encoder and decoder, consisting of 64 3x3 convolution kernels and subsequent batch normalization and ReLU activation functions.

[0108] Finally, the loss function of the model is based on Figure 3 The calculation formula is: where w k represents the weight coefficient, l (k) Represents the loss at each stage, which can be calculated using the following formula:

[0109]

[0110] in represents BCE loss (binary cross entropy loss function), represents SSIM loss (image quality loss function), Represents IoU loss (intersection over union loss function);

[0111] The calculation formula is as follows:

[0112]

[0113] Where G(r, c)∈{0, 1} represents the true label of the pixel (r, c), and S(r, c) represents the probability of predicting that the pixel is a salient point (i.e., the probability of predicting that the point is a hair area).

[0114] For SSIM loss, the predicted probability map is first split into multiple pixel blocks, represented as x = {x j :j=1,...,N 2}(image size is N×N), split the real labeled image into y={y j :j=1,...,N 2}, then the SSIM loss can be calculated according to the following formula:

[0115]

[0116] where μx 、μ y Corresponding to the mean of x and y, σ x is the variance of the mean of x, σ y is the variance of the mean of y, σ xy is the covariance of x and y, c1=0.01 2 , c1=0.03 2 ;

[0117] For loU loss, the calculation formula is as follows:

[0118]

[0119] The definitions of S(r, c) and G(r, c) are consistent with the above definitions, and H and W represent the length and width of the graph.

[0120] This example uses the BASNet algorithm to train a saliency detection model for hair regions, enabling it to correctly detect hair regions and generate a corresponding mask. The mask is then multiplied pixel by pixel with the original image to segment the hair region from the entire image for subsequent operations.

[0121] For existing technical solutions, due to the lack of strict data settings, the source domain image 1 and the reference image 2 may obey multiple different distributions respectively (taking hair color as an example, in the training samples, the original hair color may have multiple colors, and the reference hair color also has multiple colors). If both change at the same time, the final migration effect will be unstable.

[0122] This embodiment utilizes an intermediate domain conversion module to map all original domain images to the same intermediate domain, ensuring they conform to the same distribution, which can improve the migration effect to a certain extent. Specifically, in a hair color migration application, all hair colors will be converted to brown before proceeding to the next step.

[0123] This module draws on the pixel2pixel structure as the intermediate domain conversion framework. The main structure consists of two parts: the generator and the discriminator. The generator is a U-Net structure, which consists of two parts: the encoder and the decoder. Here, CK represents the Conv-BN-ReLU layer with K convolution kernels, and CDK represents the Conv-BN-Dropout-ReLU layer, where the dropout rate is 50%, the size of each convolution kernel is 4, the stride is 2, and the ratio of down-sampling to up-sampling is 2 each time. The structure of the encoder part is:

[0124] C64-C128-C256-C512-C512-C512-C512-C512;

[0125] The structure of the decoder part is:

[0126] CD512-CD1024-CD1024-C1024-C1024-C512-C256-C128.

[0127] A convolutional layer with three convolution kernels and a Tanh function is connected after the last layer of the Decoder. In addition, there is a bridge connection between the corresponding Encoder layer and the Decoder layer.

[0128] According to the previously defined rules, the discriminator structure is C64-C128-C256-C512, and the last layer is followed by a convolution operation and a sigmoid function. It is important to note that in all the above structures, the C64 convolution unit does not include the Batch Normalization operation.

[0129] In the entire model framework, the adversarial loss used can be calculated according to the following formula:

[0130] L GAN (G, D) = E y [logD(y)]+E x [log(1-D(G(x)))];

[0131] Where x represents the input image, G(x) represents the generated image, y represents the true answer image, and D(.) represents the classification probability of the discriminator. In addition to the adversarial loss, due to the presence of paired data, an additional L1 loss is used to calculate the error between the generated image 12 and the answer image. The specific formula is as follows:

[0132] L 11 =E[||yG(x)||1];

[0133] Finally, combining the generator and the discriminator, the total loss function of the model is defined as:

[0134] L total =argmin G max D L GAN (G, D) + λL l1 ;

[0135] Where λ is the weight coefficient.

[0136] The present invention utilizes the pixel2pixel model architecture to train an intermediate domain conversion model, which can convert all hair colors to the intermediate domain (i.e., yellow-brown hair color) before hair color style migration for further operation.

[0137] See also Figure 2 In this embodiment, the generation module is responsible for the actual hair color style transfer, and its structure is similar to the CycleGAN framework; there are two stages in total. In the first stage, the hair color transfer is completed and the transferred image is generated. In the second stage, the generated image is replayed to generate the image before the transfer.

[0138] For the first stage, the network contains two input branches with independent convolution modules to extract features of two images. Specifically, the structure of these two independent branches is similar to VGG, and C is used here. k Denotes a Conv-ReLU layer with K convolution kernels, and M denotes a max pooling layer. For the convolution operation, the kernel size is 3, the stride is 1, and the padding is also 1. For the pooling operation, the kernel size is 2, and the stride is 2. The structure of these two independent branches can be expressed as: C64-C64-M. After obtaining the feature maps of the two images, they are cascaded and fed into a common backbone network for feature fusion.

[0139] This backbone network, serving as the generator, also has a U-Net structure, consisting of two parts: an encoder and a decoder. The encoder's structure is similar to that of ResNet34, except that the input convolutional layer is removed and the number of convolution kernels in the first convolutional module is increased to 128. The decoder's structure is symmetrical to the encoder's, except that the downsampling operation is replaced by an upsampling operation, and an upsampling module is added at the end. Finally, to ensure that the final output feature map is compatible with both high-level and low-level features, a cascade operation is implemented between the corresponding encoder and decoder.

[0140] After the backbone network generates the final feature map, it is fed into the discriminator and then into the second stage to replay and generate the pre-migration image. The discriminator has a structure similar to ResNet18, except that the last layer is 2D instead of 1000D.

[0141] For the second stage, its structure is similar to the generator part of the first stage. The only difference is that two additional convolutional layers are added before the first convolutional module. The convolution kernel size of these two convolutional layers is 3, the stride is 1, and the padding is also 1.

[0142] In this embodiment, the supervision module is responsible for calculating part of the loss function. As described in Section 2, the present invention can process both paired and unpaired data, which is achieved by this module. Specifically, when paired data is present, the L2 loss between the generated image 12 and the answer image is calculated using the following formula.

[0143]

[0144] Where t represents the answer image, G(x) represents the generated image 12, represents the calculation of element-by-element multiplication, and M represents the hair mask of the original image. When there is no paired data, the loss function is not calculated. Therefore, the present invention can maximize the utilization of all data, including paired data and unpaired data.

[0145] There are five loss functions used in this embodiment, and the calculation formula of the total loss function is as follows:

[0146] L=l sup +l gan +l gdl +l cycle +l color ;

[0147] Among them, l sup As defined above, gan The loss is calculated as follows:

[0148] L gam (G, D) = E y [logD(y))+E x [log(1-D(G(x)))];

[0149] Where y represents the reference image and D(.) represents the classification probability of the discriminator.

[0150] For the gradient difference loss (i.e. l gdl ), its goal is to preserve the structural information of the hair to the greatest extent possible, and its calculation formula is as follows:

[0151] l gdl =l gdlx +l gdly ;

[0152] Among them, l gdlx , and l gdly Represent the gradient difference in the horizontal and vertical directions respectively, and the calculation formula is as follows:

[0153] l gdlx =||x[i,j]-x[i-1,j]|-|G(x)[i,j]-G(x)[i-1,j]||;

[0154] l gdlx =||x[i,j]-x[i,j-1]|-|G(x)[i,j]-G(x)[i,j-1]||;

[0155] Where x[i, j] represents the value of the original image x at position [i, j], and G(x)[i, j] represents the value range of the generated image 12G(x) at position [i, j]

[0156] For the circulation loss (i.e. cycle ), its goal is to maintain the global consistency information of the hair to the greatest extent possible, so it is expected that the second stage output image of the generation module, that is, the reconstructed image 11, can be as similar as possible to the input image, so the final l cycle The calculation formula is as follows:

[0157]

[0158] Where G′(G(x)) represents the output image of the second stage in the generation module for the color loss (i.e., l color ), its goal is to ensure that the hair color can be transferred to the greatest extent. To this end, the present invention uses histogram matching loss. Assuming that we want to calculate the color loss between the generated image 12G(x) and the reference image y, we first perform histogram matching on the original image x and the reference image y to obtain HM(x, y), which has the same color distribution as y but retains the same structural information as x. Then the final l color The calculation formula is as follows:

[0159]

[0160] The human style transfer method based on a generative adversarial network described in this embodiment, by adopting a reference graph framework, no longer places excessive demands on data. Both paired and unpaired data can be trained normally, greatly reducing the difficulty and cost of collecting data. Before performing style transfer, saliency segmentation is used to segment only the part that needs to be transferred, while the rest of the part remains unchanged. This embodiment designs an intermediate domain. Before performing style transfer, the source domain is first mapped to the intermediate domain, thereby overcoming the problem of multiple source domain distributions. In order to ensure the effectiveness of style transfer, this embodiment designs a corresponding loss function. Therefore, this embodiment can relatively robustly complete tasks such as human hair color transfer and human skin color transfer, and to a certain extent overcomes the problem that pixel2pixel-related methods require paired data that is difficult to meet during training, and at the same time overcomes the disadvantage that PSGAN-related methods will cause changes in some areas that are irrelevant to the transfer.

[0161] The present invention also discloses a terminal device, including a processor and a storage device, the storage device is used to store one or more programs; when the one or more programs are executed by the processor, the processor implements the above-mentioned human style transfer method based on the generative adversarial network. The so-called processor can be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components, etc. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor, etc. The so-called processor is the control center of the test equipment, and uses various interfaces and lines to connect the various parts of the entire test equipment.

[0162] The storage device can be used to store computer programs and / or modules. The processor implements various functions of the terminal device by running or executing the computer programs and / or modules stored in the storage device and calling data stored in the storage device. The storage device may mainly include a program storage area and a data storage area. The program storage area may store an operating system, at least one application required for a function, etc.; the data storage area may store data created based on the use of the terminal device, etc. In addition, the storage device may include high-speed random access memory and non-volatile memory, such as a hard disk, internal memory, a plug-in hard disk, a smart media card (SMC), a secure digital (SD) card, a flash card, at least one disk storage device, a flash memory device, or other volatile solid-state storage device.

[0163] Among them, if the module / unit integrated in the human style transfer device based on the generative adversarial network is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the present invention implements all or part of the processes in the above-mentioned embodiment method, and can also be completed by instructing the relevant hardware through a computer program. The computer program can be stored in at least one computer-readable storage medium. When the computer program is executed by the processor, it can implement the steps of the above-mentioned various method embodiments. Among them, the computer program includes computer program code, and the computer program code can be in source code form, object code form, executable file or some intermediate form. Computer-readable media may include: any entity or device that can carry computer program code, recording medium, USB flash drive, mobile hard disk, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signal, telecommunication signal and software distribution medium, etc.

[0164] It should be noted that the embodiments of the devices and apparatuses described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of the modules may be selected to achieve the objectives of the present embodiment as needed.

[0165] The specific embodiments described above further illustrate the objectives, technical solutions, and beneficial effects of the present invention. It should be understood that the above descriptions are merely specific embodiments of the present invention and are not intended to limit the scope of protection of the present invention. In particular, it should be noted that any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included within the scope of protection of the present invention for those skilled in the art.

Claims

1. A human style transfer method based on generative adversarial network, characterized in that: Style transfer applied to human body parts, including: Identify the region where the migration target is located in the source domain image and output the region as the first segmentation target; Identify the region where the migration target is located in the reference image, and output the region as the second segmentation target; Mapping the first segmentation target to the first intermediate domain using an intermediate domain conversion module to obtain a third segmentation target; wherein the intermediate domain conversion model is trained using a pixel2pixel model architecture to map the source domain image to the same intermediate domain so that they obey the same distribution; Performing style transfer on the third segmentation target based on the second segmentation target to generate a first map; Covering the first map to the area of the first segmentation target in the source domain image, and outputting the covered source domain image as a generated image; Before outputting the covered source domain image as the generated image, it also includes: Verifying the style transfer effect of the generated image relative to the source domain image and the reference image through a total loss function; wherein the total loss function includes an adversarial loss function, a gradient difference loss function, a cycle loss function, and a color loss function; The adversarial loss function is used to verify the distribution difference between the first map and the second segmentation target; The gradient difference loss function is used to prioritize the structural information of the migration target; The loop loss function is used to prioritize the global consistency information of the migration target; The color loss function is used to prioritize the color information of the migration target; When the source domain image and the reference image are paired, the total loss function also includes a minimum square error loss function; the minimum square error loss function is used to calculate the L2 loss between the generated image and the second segmentation target.

2. The human body style transfer method based on generative adversarial network according to claim 1, characterized in that: The method further comprises: identifying the region where the migration target is located in the source domain image and outputting the region as the first segmentation target; identifying the region where the migration target is located in the reference image and outputting the region as the second segmentation target; specifically: Establish a segmentation model based on the BASNet saliency detection algorithm; Collect images containing the migration target to establish a first training set and a first test set; Training and testing the segmentation model using the first training set and the first test set; Inputting the source domain image into a segmentation model and outputting a first segmentation target; The reference image is input into a segmentation model, and a second segmentation target is output.

3. The human body style transfer method based on generative adversarial network according to claim 1, characterized in that: The first segmentation target is mapped onto the first intermediate domain to obtain a third segmentation target, specifically: Based on the pixel2pixel model architecture, an intermediary domain conversion model is established; Collect images containing the first segmented target to establish a second training set and a second test set; Training and testing the intermediate domain conversion model using the second training set and the second test set; The first segmentation target is mapped to the first intermediate domain through an intermediate domain conversion model to obtain a third segmentation target.

4. A human style transfer device based on generative adversarial network, characterized in that: Executing the human body style transfer method according to any one of claims 1 to 3, the human body style transfer device comprises: a first segmentation module, a second segmentation module, an intermediate domain conversion module, a migration module, a generation module and a verification module; The first segmentation module is used to identify the region where the migration target is located in the source domain image, and output the region as a first segmentation target; The second segmentation module is used to identify the area where the migration target is located in the reference image, and output the area as a second segmentation target; The intermediate domain conversion module is used to map the first segmentation target to the first intermediate domain to obtain a third segmentation target; The migration module is used to perform style migration on the third segmentation target based on the second segmentation target to generate a first map; The generating module is configured to cover the first map to the area of the first segmentation target in the source domain image, and output the covered source domain image as a generated image; The human body style transfer device further includes a verification module; The verification module is used to verify the style transfer effect of the generated image relative to the source domain image and the reference image through the total loss function; The verification module further includes a first judgment unit and a minimum square error loss unit; The first judging unit is used to judge whether the source domain image and the reference image are paired; The least square error loss unit is used to calculate the L2 loss between the generated image and the second segmentation target when the first judgment unit determines that the result is yes.

5. The human body style transfer device based on generative adversarial network according to claim 4, characterized in that: Also includes a verification module; The verification module is used to verify the style transfer effect of the generated image relative to the source domain image and the reference image through the total loss function; The verification module includes an adversarial loss unit, a gradient difference loss unit, a cycle loss unit and a color loss unit; The adversarial loss unit is used to verify the distribution difference between the first map and the second segmentation target; The gradient difference loss unit is used to prioritize the structural information of the migration target; The cycle loss unit is used to prioritize maintaining global consistency information of the migration target; The color loss unit is used to preferentially maintain color information of the migration target.

6. A computer-readable storage medium, characterized in that The computer-readable storage medium includes a stored computer program, wherein, when the computer program is running, the device where the computer-readable storage medium is located is controlled to execute the human style transfer method based on a generative adversarial network according to any one of claims 1 to 4.

Images