Method for generating portrait color image through portrait sketch based on improved CycleGAN
The symmetric GELAN architecture in CycleGAN enhances feature extraction to generate higher-quality color facial images from sketches, addressing challenges in image detail recovery and model stability.
Patent Information
- Application Number
- CN202510350025.2
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-03-24
- Publication Date
- 2025-07-15
AI Technical Summary
When generating face sketches and generating face color pictures, the generated image details are missing, the model training is difficult to converge, and the gradient between the generator and the discriminator disappears or explodes, resulting in unsatisfactory image quality.
The cyclic generation adversarial network is adopted with the generator of SGELAN. By constructing a deep feature extraction module of a symmetric GELAN structure, combined with the PatchGAN discriminator, the cyclic consistency loss of CycleGAN is used for adversarial training to realize the conversion of portrait sketch to portrait color pictures.
Generate higher quality and more detailed color face images, which improves the training stability of the model and the authenticity and consistency of the generated images.
Smart Images
Figure CN120318350A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of deep learning and neural networks, and particularly relates to a method for generating a portrait color image from a human pixel sketch based on an improved CycleGAN. Background Art
[0002] Generating a corresponding color face image from a face sketch is one of the most important research fields in computer vision and deep learning, providing important technical support for fields such as face recognition, digital art, medical imaging, virtual character design, and security monitoring. Although in the aspect of generating sketch-to-image conversion, deep learning methods, especially generative adversarial networks, have made certain progress, generating a realistic color face image from a black-and-white sketch image still faces numerous challenges. The main difficulties include: how to recover rich details, colors, and textures from the simplified sketch information; how to handle the diversity and individuality of facial features, how to ensure the authenticity and consistency of the generated image, and how to cope with the limitations of computing resources and training data. These problems still need to be better solved.
[0003] In recent years, with the development of deep learning and neural network technologies, the technology of generating a color face image from a face sketch has achieved certain results. Especially, the generative adversarial network (GAN) has achieved the most outstanding results in this task. For example, the image-to-image generative adversarial network (Pix2Pix GAN) introduced in 2016 introduced the structure of U-Net into the generator, enabling the generator to learn the implicit relationship between images and corresponding domains, and realizing the transformation from one image domain to another. However, due to only one generator for training and learning, the quality of the finally generated image is not ideal. CycleGAN in 2017 solved this problem well. It adopted two sets of generative adversarial network models and used cyclic consistency constraints to make the models influence and optimize each other, thereby generating higher-quality images. However, since the training process of CycleGAN includes two generators and two discriminators, the training process of the model is more complex and unstable than that of ordinary GAN. The game process between the generator and the discriminator is prone to problems such as vanishing gradients or exploding gradients, making it difficult for the model to converge. On the other hand, one of the core innovations of CycleGAN is the "cyclic consistency" principle, that is, an image should be restored to the original image after two conversions. However, this loss function has a high dependence on the training of the model. If the conversion relationships of some images in the dataset are relatively complex or difficult to maintain cyclic consistency, this may cause the model to fail to learn effective conversion relationships, thereby affecting the quality of the generated results.
[0004] Therefore, there are still many problems to be solved in the field of generating a portrait color image from a human pixel sketch. Such as missing details in the generated image and difficult convergence of model training. Summary of the Invention
[0005] In view of the deficiencies of the prior art, the present invention uses a cyclic generative adversarial network with a generator containing SGELAN (symmetric GELAN architecture), enabling the model to extract image features even at deeper network levels, thereby generating images of higher quality and more details.
[0006] To achieve the above object, the present invention provides a method for generating portrait color images from human pixel sketches based on an improved CycleGAN, comprising the following steps:
[0007] (1) Collect human pixel sketches and corresponding portrait color images, and perform preprocessing to construct a dataset for generating portrait color images from human pixel sketches;
[0008] (2) Construct a first generator for converting portrait color images into human pixel sketches and a second generator for converting human pixel sketches into portrait color images. The first generator and the second generator have the same model, both including an encoder module, a deep feature extraction module, and a decoder module; wherein the backbone network SGELAN of the deep feature extraction module is a symmetric GELAN structure;
[0009] (3) Construct a first discriminator for discriminating the probability that an image is a portrait color image and a second discriminator for discriminating the probability that an image is a human pixel sketch;
[0010] (4) Based on the first generator, the second generator, the first discriminator, and the second discriminator, form a cyclic generative adversarial network CycleGAN. Learn according to the dataset for generating portrait color images from human pixel sketches, and use the cyclic consistency loss of CycleGAN to connect the two generators for converting portrait color images into human pixel sketches and converting human pixel sketches into portrait color images, and perform adversarial training to realize generating portrait color images from human pixel sketches.
[0011] Further, the preprocessing is specifically as follows:
[0012] Perform data augmentation processing on the data through random cropping, random horizontal flipping, and random rotation;
[0013] Trim all data to the same size and perform normalization processing.
[0014] Further, in the deep feature extraction module of the first generator and the second generator models, the backbone network SGELAN is obtained by deforming GELAN. After the input feature map is evenly divided into two parts, the processing method of the first part is the same as that of the second part. Also perform feature extraction processing through multiple layers of convolution, and finally serially combine the features extracted by each layer of convolution for output;
[0015] The deep feature extraction module stacks 4 layers of SGELAN. The number of input feature channels for each layer is 256, and the output feature channels are 512. Between every two layers of SGELAN, the number of feature channels is reduced to 256 through one-dimensional convolution to reduce the number of model parameters, and finally deep feature extraction is achieved.
[0016] Furthermore, the encoder modules in the first generator and the second generator models are implemented by combining a convolutional layer with a convolution kernel size of 7 and a stride of 1 and a convolutional layer with a convolution kernel size of 3 and a stride of 2 as a group, and stacking three groups.
[0017] Furthermore, the decoder modules in the first generator and the second generator models use a transposed convolution with a convolution kernel size of 3 and a stride of 2, stack three layers, and finally output the target image through a two-dimensional convolution with a convolution kernel size of 7.
[0018] Furthermore, the first discriminator and the second discriminator models are the same and are obtained based on PatchGAN; the backbone network uses 5 convolutional layers. The convolution kernel sizes of the first three convolutional layers are 4 and the stride is 2, and the convolution kernel size of the fourth convolutional layer is 4 and the stride is 1; the initial input feature channel number of the discriminator model is 3, the output feature channel amount of the first layer is 64, the output feature channel amounts of the subsequent 3 convolutional networks double layer by layer, and the output feature channel number of the last convolutional network is 1, and a global pooling is used to obtain the score.
[0019] Furthermore, the specific content of step (4) is as follows:
[0020] (4.1) Input the image of the human pixel sketch generating the portrait color image dataset into the CycleGAN;
[0021] (4.2) Calculate the cycle consistency loss;
[0022] Loss cycle = E|G x (G y (x)) - x| + E|G y (G x (y)) - y|, (x ∈ X, y ∈ Y)
[0023] (4.3) Calculate the GAN loss;
[0024] Loss GAN (G x , D x ) = E[log(D x (x))] + E[log(1 - D x (G x (x)))]
[0025] (4.4) Calculate the total loss:
[0026] Loss full = Loss GAN (G x , D x ) + Loss GAN (G y , D y ) + λLoss cycle
[0027] Where: G x represents the first generator, G y represents the second generator, D x represents the first discriminator, D y represents the second discriminator, and λ is the weight coefficient of the cycle consistency loss;
[0028] (4.5) Backpropagate the calculated loss values of various types to obtain the gradient values of the corresponding parameters of each model;
[0029] (4.6) All generator models and discriminator models select the Adam optimizer and execute the corresponding parameter update algorithm;
[0030] (4.7) Repeat steps (4.1) to (4.6) until the loss value converges or reaches the set maximum number of cycles.
[0031] The present invention also provides a device for generating a colored portrait from a human pixel sketch based on an improved CycleGAN, including:
[0032] A data acquisition module for collecting human pixel sketches and corresponding colored portraits, performing preprocessing, and constructing a dataset for generating colored portraits from human pixel sketches;
[0033] A generator construction module for constructing a first generator for converting a colored portrait into a human pixel sketch and a second generator for converting a human pixel sketch into a colored portrait. The first generator and the second generator have the same model, and both include an encoder module, a deep feature extraction module, and a decoder module; where the backbone network SGELAN of the deep feature extraction module is a symmetric GELAN structure;
[0034] A discriminator construction module for constructing a first discriminator for discriminating the probability that an image is a colored portrait and a second discriminator for discriminating the probability that an image is a human pixel sketch;
[0035] A processing module is used to form a Cycle Generative Adversarial Network (CycleGAN) based on the first generator, the second generator, the first discriminator, and the second discriminator. It generates a portrait color image dataset based on the human pixel sketch for learning, and uses the cycle consistency loss of CycleGAN to connect the two generators for converting portrait color images to human pixel sketches and human pixel sketches to portrait color images, and conducts adversarial training to realize the generation of portrait color images from human pixel sketches.
[0036] The present invention also provides an electronic device, including: at least one processor; and a memory communicatively connected to the at least one processor; the memory stores instructions executable by the at least one processor, and when the instructions are executed by the at least one processor, the at least one processor is enabled to execute the above method.
[0037] The present invention also provides a non-transitory computer-readable storage medium storing computer instructions for causing a computer to execute the above method.
[0038] Advantages of the present invention:
[0039] In the deep feature extraction module of the generator of the present invention, a Symmetric-based GELAN (SGELAN) module is adopted. This module integrates many advantages of modules such as ResNet, RepVGG, and CSP, enabling the model to extract image features even at deeper network levels, thereby generating higher-quality and more detailed images. BRIEF DESCRIPTION OF THE DRAWINGS
[0040] Figure 1 It is a schematic flowchart of a method for generating portrait color images from human pixel sketches based on an improved CycleGAN according to an embodiment of the present invention.
[0041] Figure 2 It is a schematic structural diagram of a GELAN module according to an embodiment of the present invention.
[0042] Figure 3 It is a schematic structural diagram of an SGELAN module according to an embodiment of the present invention.
[0043] Figure 4 It is a schematic diagram of a CycleGAN network architecture according to an embodiment of the present invention. DETAILED DESCRIPTION OF THE EMBODIMENTS
[0044] The present invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be noted that the following embodiments are only for further illustration of the present invention and should not be construed as limiting the protection scope of the present invention. Persons skilled in the relevant art can make some non-essential improvements and adjustments to the present invention based on the above inventive content and still fall within the protection scope of the present invention when implementing it specifically.
[0045] As shown Figure 1 in the figure, the present invention provides a method for generating portrait color images from human pixel sketches based on an improved CycleGAN, including the following steps:
[0046] S101. Collect human pixel sketches and corresponding portrait color images, and construct a dataset for generating portrait color images from human pixel sketches.
[0047] The present invention uses the open-source CUFS dataset and hires sketch artists to draw some sketches and corresponding color images to expand the data volume.
[0048] Perform data augmentation processing on the above data, such as random cropping, random horizontal flipping, random rotation, etc.
[0049] Trim all data to the same size and perform normalization to ensure the consistency of the input data.
[0050] Use the preprocessed data as the dataset for generating portrait color images from human pixel sketches. Among them, the human pixel sketch is used as the input domain X of the model, and the corresponding portrait color image is used as the output domain Y.
[0051] S102. Construct a first generator for converting portrait color images into human pixel sketches and a second generator for converting human pixel sketches into portrait color images. The first generator and the second generator have the same model structure, both including an encoder module, a deep feature extraction module, and a decoder module. The backbone network SGELAN of the deep feature extraction module is a symmetric GELAN structure.
[0052] The first generator G x and the second generator G y The model architecture is mainly the UNet architecture, which includes three main modules, namely an encoder module, a deep feature extraction module, and a decoder module.
[0053] The deep feature extraction module, whose backbone network SGELAN is obtained by deforming GELAN (symmetric GELAN), such as Figure 2 , Figure 3As shown in the figure, after the input feature map is evenly divided into two parts, GELAN extracts the lower-layer features of the second part through multiple layers of convolution while retaining the upper-layer features of the first part, and finally serially combines the features extracted by each layer of convolution. While retaining the upper-layer features of the first part, SGELAN performs the same feature extraction process on the upper-layer features as the second part, that is, performs feature extraction through multiple layers of convolution, and finally serially combines the features extracted by each layer of convolution for output. The deep feature extraction module stacks 4 layers of SGELAN, with the input feature channel amount of 256 for each layer and the output feature channel of 512, and reduces the feature channel amount to 256 through one-dimensional convolution between every two layers of SGELAN to reduce the model parameter amount, and finally realizes the extraction of deep features.
[0054] The encoder module is implemented by stacking three groups, each group combining a convolutional layer with a convolutional kernel size of 7 and a stride of 1 and a convolutional layer with a convolutional kernel size of 3 and a stride of 2.
[0055] The decoder module uses a transposed convolution with a convolutional kernel size of 3 and a stride of 2, stacks three layers, and finally outputs the target image through a two-dimensional convolution with a convolutional kernel size of 7.
[0056] S103. Construct a first discriminator for discriminating the probability that the image is a portrait color picture and a second discriminator for discriminating the probability that the image is a portrait sketch.
[0057] The first discriminator D x and the second discriminator D y have the same model, which is mainly obtained by deformation on the basis of PatchGAN. The backbone network uses 5 convolutional layers. The convolutional kernel sizes of the first three convolutional layers are 4 and the stride is 2, and the convolutional kernel size of the fourth convolutional layer is 4 and the stride is 1; the initial input feature channel number of the discriminator model is 3, the output feature channel amount of the first layer is 64, the output feature channel amounts of the subsequent 3 convolutional networks double layer by layer, and the output feature channel number of the last convolutional network is 1, and a global pooling is used to obtain the score.
[0058] S104. Based on the first generator, the second generator, the first discriminator and the second discriminator, form a CycleGAN, learn according to the portrait sketch to generate a portrait color picture dataset, and use the cycle consistency loss of CycleGAN to connect the two generators of converting the portrait color picture to the portrait sketch and converting the portrait sketch to the portrait color picture, and perform adversarial training to realize generating portrait color pictures from portrait sketches.
[0059] As Figure 4 shown, in the CycleGAN, the first generator G x is mainly responsible for converting the image from the image domain Y (portrait color picture) to the image domain X (portrait sketch), and the second generator Gy It is mainly responsible for converting the image from the image domain X to the image domain Y. The first discriminator D of the discriminator x and the second discriminator D y are respectively used to discriminate the probability that the image belongs to the image domain X and the probability that the image belongs to the image domain Y.
[0060] Based on the above cycle generative adversarial network, the specific training process is as follows:
[0061] (1) Input the human pixel sketch generated portrait color map dataset image into the cycle generative adversarial network CycleGAN.
[0062] (2) Calculate the cycle consistency loss of the model.
[0063] Loss cycle = E|G x (G y (x)) - x| + E|G y (G x (y)) - y|, (x ∈ X, y ∈ Y)
[0064] (3) Calculate the GAN loss of the model. For the generator G x and the discriminator.
[0065] Loss GAN (G x , D x ) = E[log(D x (x))] + E[log(1 - D x (G x (x)))]
[0066] (4) Calculate the total loss:
[0067] Loss full = Loss GAN (G x , D x ) + Loss GAN (G y , D y ) + λLoss cycle
[0068] Among them: G x represents the first generator, G y represents the second generator, D x represents the first discriminator, D y represents the second discriminator, and λ is the weight coefficient of the cycle consistency loss.
[0069] (5) Perform backpropagation on the calculated various loss values to obtain the gradient values of the corresponding parameters of each model.
[0070] (6) All generator models and discriminator models select the Adam optimizer and execute the corresponding parameter update algorithm.
[0071] (7) Repeat steps (1) to (6) until the loss value converges or reaches the set maximum number of iterations.
[0072] An embodiment of the present invention also provides a device for generating a color portrait from a human pixel sketch based on an improved CycleGAN, including:
[0073] A data acquisition module for acquiring human pixel sketches and corresponding color portraits of human figures, performing preprocessing, and constructing a dataset for generating color portraits of human figures from human pixel sketches;
[0074] A generator construction module for constructing a first generator for converting a color portrait of a human figure into a human pixel sketch and a second generator for converting a human pixel sketch into a color portrait of a human figure. The first generator and the second generator have the same model, and both include an encoder module, a deep feature extraction module, and a decoder module; wherein the backbone network SGELAN of the deep feature extraction module is a symmetric GELAN structure;
[0075] A discriminator construction module for constructing a first discriminator for discriminating the probability that an image is a color portrait of a human figure and a second discriminator for discriminating the probability that an image is a human pixel sketch;
[0076] A processing module for forming a cyclic generative adversarial network CycleGAN based on the first generator, the second generator, the first discriminator, and the second discriminator, learning according to the dataset for generating color portraits of human figures from human pixel sketches, and using the cyclic consistency loss of CycleGAN to connect the two generators for converting a color portrait of a human figure into a human pixel sketch and converting a human pixel sketch into a color portrait of a human figure, and performing adversarial training to realize generating a color portrait of a human figure from a human pixel sketch.
[0077] An embodiment of the present invention also provides an electronic device, including: at least one processor; and a memory communicatively connected to the at least one processor; the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to execute the above method.
[0078] An embodiment of the present invention also provides a non-transitory computer-readable storage medium storing computer instructions for causing a computer to execute the above method.
[0079] The specific description above further elaborates on the purpose, technical solution, and beneficial effects of the invention. It should be understood that the above is only a specific embodiment of the present invention and is not used to limit the protection scope of the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principle of the present invention shall be included within the protection scope of the present invention.
Claims
1. A method for generating portrait color images from human pixel sketches based on improved CycleGAN, characterized in that, It includes the following steps: (1) Collect human pixel sketches and corresponding human portrait color images, perform preprocessing, and construct a human pixel sketch to human portrait color image dataset; (2) Construct a first generator for converting human portrait color images to human pixel sketches and a second generator for converting human pixel sketches to human portrait color images. The first generator and the second generator have the same model, both including an encoder module, a deep feature extraction module, and a decoder module; among them, the backbone network SGELAN of the deep feature extraction module is a symmetric GELAN structure; (3) Construct a first discriminator for discriminating the probability that an image is a human portrait color image and a second discriminator for discriminating the probability that an image is a human pixel sketch; (4) Based on the first generator, the second generator, the first discriminator, and the second discriminator, form a Cycle Generative Adversarial Network (CycleGAN). Learn according to the human pixel sketch to human portrait color image dataset. Use the cycle consistency loss of CycleGAN to connect the two generators of converting human portrait color images to human pixel sketches and converting human pixel sketches to human portrait color images, and perform adversarial training to realize generating human portrait color images from human pixel sketches.
2. The method for generating a portrait color image from a human pixel sketch based on the improved CycleGAN according to claim 1, wherein The specific preprocessing is as follows: Perform data augmentation processing on the data through random cropping, random horizontal flipping, and random rotation; Trim all data to the same size and perform normalization processing.
3. The method for generating a portrait color picture from a human pixel sketch based on the improved CycleGAN according to claim 1, wherein, In the deep feature extraction module of the first generator and the second generator models, the backbone network SGELAN is obtained by deforming GELAN. After the input feature map is evenly divided into two parts, the processing method of the first part is the same as that of the second part. Also perform feature extraction processing through multiple layers of convolution, and finally serially combine the features extracted by each layer of convolution for output; The deep feature extraction module stacks 4 layers of SGELAN. The input feature channel amount of each layer is 256, and the output feature channel is 512. And between every two layers of SGELAN, reduce the feature channel amount to 256 through one-dimensional convolution to reduce the number of model parameters, and finally realize the extraction of deep features.
4. The method for generating a portrait color picture from a human pixel sketch based on the improved CycleGAN according to claim 1, wherein The encoder module in the first generator and the second generator models is realized by combining a convolutional layer with a convolution kernel size of 7 and a stride of 1 and a convolutional layer with a convolution kernel size of 3 and a stride of 2 as a group, and stacking three groups.
5. The method for generating a human portrait color picture from a human pixel sketch based on the improved CycleGAN according to claim 1, wherein The decoder module in the first generator and the second generator models uses a transposed convolution with a convolution kernel size of 3 and a stride of 2, stacks three layers, and finally outputs the target image through a two-dimensional convolution with a convolution kernel size of 7.
6. The method for generating a human portrait color picture from a human pixel sketch based on the improved CycleGAN according to claim 1, characterized in that: The first discriminator and the second discriminator models are the same and are obtained based on PatchGAN; the backbone network adopts 5 layers of convolutional layers. The convolution kernel sizes of the first three convolutional layers are 4 and the stride is 2. The convolution kernel size of the fourth convolutional layer is 4 and the stride is 1; the initial input feature channel number of the discriminator model is 3, the output feature channel amount of the first layer is 64, and the output feature channel amounts of the subsequent 3 layers of convolutional networks are doubled layer by layer. The output feature channel number of the last layer of convolutional network is 1, and a global pooling is used to obtain the score.
7. The method for generating a color portrait from a human pixel sketch based on the improved CycleGAN according to claim 1, wherein The specific step (4) is as follows: (4.1) Input the images of the human pixel sketch to human portrait color image dataset into the Cycle Generative Adversarial Network (CycleGAN); (4.2) Calculate the cycle consistency loss; Loss cycle = E|G x (G y (x)) - x| + E|G y (G x (y)) - y|, (x ∈ X, y ∈ Y) (4.3) Calculate the GAN loss; Loss GAN (G x ,D x ) = E[log(D x (x))] + E[log(1 - D x (G x (x)))] (4.4) Calculate the total loss: Loss full = Loss GAN (G x , D x ) + Loss GAN (G y , D y ) + λLoss cycle Among them: G x represents the first generator, G y represents the second generator, D x represents the first discriminator, D y represents the second discriminator, and λ is the weight coefficient of the cycle consistency loss; (4.5) Perform backpropagation on the calculated loss values of various types to obtain the gradient values of the corresponding parameters of each model; (4.6) For all generator models and discriminator models, select the Adam optimizer and execute the corresponding parameter update algorithm; (4.7) Repeat steps (4.1) to (4.6) until the loss value converges or reaches the set maximum number of cycles.
8. A device for generating human portrait color pictures from human pixel sketches based on an improved CycleGAN, characterized in that, It includes: A data acquisition module for acquiring human pixel sketches and corresponding portrait color images, preprocessing them, and constructing a dataset for generating portrait color images from human pixel sketches; A generator construction module for constructing a first generator for converting portrait color images into human pixel sketches and a second generator for converting human pixel sketches into portrait color images. The first generator and the second generator have the same model, and both include an encoder module, a deep feature extraction module, and a decoder module; among them, the backbone network SGELAN of the deep feature extraction module is a symmetric GELAN structure; A discriminator construction module for constructing a first discriminator for discriminating the probability that an image is a portrait color image and a second discriminator for discriminating the probability that an image is a human pixel sketch; A processing module for forming a Cycle Generative Adversarial Network CycleGAN based on the first generator, the second generator, the first discriminator, and the second discriminator, learning according to the dataset for generating portrait color images from human pixel sketches, using the cycle consistency loss of CycleGAN to connect the two generators for converting portrait color images into human pixel sketches and converting human pixel sketches into portrait color images, and performing adversarial training to realize generating portrait color images from human pixel sketches.
9. An electronic device, comprising: At least one processor; And a memory communicatively connected to the at least one processor; characterized in that the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the method according to any one of claims 1-7.
10. A non-transitory computer-readable storage medium storing computer instructions, characterized in that, The computer instructions are used to cause the computer to execute the method according to any one of claims 1-7.