A hairstyle transplantation method based on generative adversarial network GAN

Abstract

A hairstyle transplantation method based on a generative adversarial network (GAN) belongs to the field of image processing and computer vision, and includes the following steps: S1, training a generative adversarial network model; S2, using image segmentation technology to generate masks of corresponding areas of a source image and a target image; S3, scaling and moving the source image to align the hairstyle area of ​​the source image with the hairstyle area of ​​the target image; S4, adjusting the color distribution of the source image to make it close to the color distribution of the target image; S5, inputting the source image, the target image and the related masks into the model to fuse the hairstyle area of ​​the target image with the head area of ​​the source image; S6, adjusting the details of the fused image. The hairstyle transplantation method based on a generative adversarial network (GAN) provided by the present invention can align the hairstyle with the target image and match the hairstyle color, solves the problem that the effect after hairstyle transplantation is not natural enough, and improves the effect of hairstyle transplantation.

Description

Technical Field

[0001] The present invention belongs to the field of image processing and computer vision, and in particular relates to a hairstyle transplantation method based on a generative adversarial network (GAN). Background Art

[0002] Hairstyle transplantation is a technology that processes the hair styles of people in pictures to achieve the effect of changing the hairstyles of people. This satisfies people's needs to preview whether a certain hairstyle is suitable for them in advance, and people can easily see what effect will be achieved if they change their hairstyles. However, traditional image processing methods often perform poorly in complex image editing scenarios. For example, when performing hairstyle transplantation using traditional methods, there are usually problems with shape alignment and color matching in hairstyle transplantation, which makes the effect of hairstyle transplantation unnatural. For example, the shape of the hairstyle is not aligned with the target image, and the color of the hairstyle is inconsistent with the target image.

[0003] Generative adversarial networks (GANs), as an important technology in the field of deep learning, can generate realistic images through training, thereby achieving a more natural image fusion effect. GANs have been widely used in tasks such as image generation and style transfer. Through adversarial training, they can generate synthetic images that are difficult to distinguish from real images, providing a new solution for image editing. Therefore, in order to solve the shape alignment and color matching problems in the above-mentioned hairstyle transplantation process, it is necessary to establish a new hairstyle transplantation method through GANs. The present invention solves this technical problem. Summary of the invention

[0004] The present invention provides a hairstyle transplantation method based on a generative adversarial network (GAN), which can solve the problem that the effect of hairstyle transplantation is not natural enough, and can align the hairstyle with the target image and match the color of the hairstyle with the color of the target image, thereby improving the effect of hairstyle transplantation.

[0005] A hairstyle transplantation method based on a generative adversarial network (GAN) comprises the following steps:

[0006] S1. Train the generative adversarial network model to make the generated images more realistic;

[0007] S2, using image segmentation technology to generate masks of corresponding areas of the source image and the target image to ensure that only relevant areas are processed during the hair transplantation process;

[0008] S3, scaling and moving the source image so that the hairstyle area of ​​the source image is aligned with the hairstyle area of ​​the target image;

[0009] S4, adjusting the color distribution of the source image to make it close to the color distribution of the target image;

[0010] S5, inputting the source image, the target image and the related mask into the model so that the model can fuse the hairstyle region of the target image with the head region of the source image, thereby realizing the hairstyle transplantation;

[0011] S6. Adjust the details of the fused image.

[0012] Furthermore, the step S2 includes the following steps:

[0013] S21, preprocess the image I through Processor to meet the input requirements of the Segformer model, expressed as:

[0014] ;

[0015] Among them, X represents the preprocessed image, which is used as the input of the model;

[0016] S22. Input the preprocessed image into the Segformer model. The mathematical expression is:

[0017] ;

[0018] S23. Use the Segformer algorithm to segment the image to generate a mask containing the hair area.

[0019] Furthermore, the step S3 includes the following steps:

[0020] S31, defining a hairstyle region as a region of interest;

[0021] S32, the boundary rectangle of the source image contour is expressed as:

[0022] ;

[0023] The bounding rectangle of the target image contour is expressed as:

[0024] ;

[0025] in, Indicates the coordinates of the upper left corner of the region of interest in the source image; Represents the width of the region of interest in the source image, that is, the number of pixels in the horizontal direction; Represents the height of the region of interest in the source image, that is, the number of pixels in the vertical direction; Indicates the coordinates of the upper left corner of the region of interest in the target image; Indicates the width of the region of interest in the target image; Indicates the height of the region of interest in the target image;

[0026] S33, the aligned image is calculated by the following formula

[0027] ;

[0028] in, Represents the scaled source image; Represents the scaling operation of an image; represents the original source image; Represents the width scaling factor, which is used to adjust the width of the source image to match the width of the target image; Represents the height scaling factor, which is used to adjust the height of the source image to match the height of the target image;

[0029] ;

[0030] in, Represents the adjusted source image; Translate represents the translation operation of the image; Indicates the horizontal translation of the source image, which is used to align the aligned image horizontally with the region of interest of the target image; Indicates the vertical translation of the source image, which is used to align the aligned image vertically with the region of interest of the target image.

[0031] Further, the step S4 includes the following steps:

[0032] S41. Calculate source image Color distribution histogram , and calculate the target image Color distribution histogram , where v represents the color value;

[0033] S42, calculating the cumulative distribution function of the source image and the target image;

[0034] S43. For each color value in the source image , find the color value closest to its cumulative distribution function in the target image , and Mapping , get the color-matched image .

[0035] Furthermore, in step S42, the cumulative distribution function of the source image is calculated by the following formula:

[0036] ;

[0037] The cumulative distribution function of the target image is calculated by the following formula:

[0038] .

[0039] Furthermore, in step S43, the color value of the source image is Maps to the closest color value in the target image The process is expressed by the following formula:

[0040] ;

[0041] The color distribution of the source image is adjusted in the above manner to make it close to the color distribution of the target image.

[0042] Furthermore, the specific steps of step S5 are as follows:

[0043] S51. In the generation phase of GAN, the model accepts the input source image, target image and related masks, and generates the fused image through the generator. ;

[0044] S52, evaluating the image through the discriminator D, which is used to distinguish the real image x from the generated image G(z);

[0045] S53, through the optimization process, gradually adjust the parameters of the generator G and the discriminator D to make the generated image close to the real effect;

[0046] S54. The above content is expressed by the following formula:

[0047] ;

[0048] in, represents the source image, A mask representing the hair, represents the target hairstyle image, and z represents random noise.

[0049] Furthermore, the optimization process in step S53 is as follows:

[0050] S531, using gradient descent to update the parameters of the generator and the discriminator;

[0051] S532, the method of updating the discriminator parameters is as follows:

[0052] ;

[0053] in, represents the parameters of the discriminator; represents the learning rate; Represents the loss function of the discriminator About the discriminator parameters The gradient of

[0054] S533. The method of updating the generator parameters is as follows:

[0055] ;

[0056] in, Represents the parameters of the generator; Represents the loss function of the generator About generator parameters gradient.

[0057] Furthermore, the adversarial training process in step S1 is as follows:

[0058] S11. Generate a synthetic image that is consistent with the style of the input image through the generator, so that the generated image is as realistic as possible, so that it is difficult to distinguish from the real image. The loss function of the generator is expressed by the following formula:

[0059] ,

[0060] in, Represents the loss function of the generator. The purpose of the generator is to minimize the discriminator D on the generated image The recognition result makes Approaching 1, the generated image is close to the real image;

[0061] S12. The discriminator is used to distinguish between real images and generated images. The purpose is to maximize the accurate distinction between real images and generated images, that is, real images are judged as real as much as possible, and generated images are judged as false as much as possible. The loss function of the discriminator is expressed by the following formula:

[0062] ,

[0063] in, represents the loss function of the discriminator, represents the ability of the discriminator to correctly identify real images, Represents the ability of the discriminator to correctly identify generated images as fake;

[0064] S13. As training progresses, the generator gradually learns to imitate the details of real images, making the images generated by the generator increasingly difficult to distinguish from real images.

[0065] Furthermore, step S6 includes adjusting the picture details through picture editing software to achieve the most natural visual effect.

[0066] The technical effects of the present invention are as follows:

[0067] (1) This scheme generates images through a generative adversarial network (GAN) to make the subsequently generated images more realistic. The mask of the corresponding part is determined through image segmentation technology. The source image is then adjusted so that the hairstyle area of ​​the source image is close to the hairstyle area of ​​the target image in terms of position and color. The trained GAN model is then used to generate a hairstyle fused image, that is, an image in which the hairstyle of the target image is fused with the head of the source image. This achieves a hairstyle transplantation effect and improves the visual authenticity of the generated image.

[0068] (2) This scheme uses image segmentation technology to generate masks of the corresponding parts of the source image and the target image, such as masks of the hairstyle area and the head area, so as to ensure that only the relevant areas are processed during the transplantation process and avoid affecting other areas;

[0069] (3) This scheme can align the hairstyle area of ​​the source image with the hairstyle area of ​​the target image by scaling and translating the source image, and make the color of the hairstyle area of ​​the source image the same as the color of the hairstyle area of ​​the target image through color histogram and other methods, which is convenient for subsequent hairstyle transplantation operations;

[0070] (4) Generative adversarial networks (GANs) can improve the realism of images generated by the generator through adversarial training, and can continuously optimize the parameters of the generator and discriminator during the training process, so that the generated fused images gradually approach the real effect, thereby improving the authenticity of the hairstyle transplantation images. BRIEF DESCRIPTION OF THE DRAWINGS

[0071] Figure 1 It is a schematic diagram of the process of the present invention. DETAILED DESCRIPTION

[0072] The technical solution of the present invention will be clearly and completely described below in conjunction with specific embodiments and drawings.

[0073] See also Figure 1 , a hairstyle transplantation method based on a generative adversarial network (GAN), comprising the following steps:

[0074] S1. Train the generative adversarial network model to make the generated images more realistic;

[0075] S2, using image segmentation technology to generate masks of corresponding areas of the source image and the target image to ensure that only relevant areas are processed during the hair transplantation process;

[0076] S3, scaling and moving the source image so that the hairstyle area of ​​the source image is aligned with the hairstyle area of ​​the target image;

[0077] S4, adjusting the color distribution of the source image to make it close to the color distribution of the target image;

[0078] S5, inputting the source image, the target image and the related mask into the model so that the model can fuse the hairstyle region of the target image with the head region of the source image, thereby realizing the hairstyle transplantation;

[0079] S6. Adjust the details of the fused image.

[0080] The mask part may include a mask of the hairstyle area and a mask of the head (face) area, so as to facilitate the subsequent transplantation of the hairstyle of the target image to the source image.

[0081] Further, step S2 includes the following steps:

[0082] S21, preprocess the image I through Processor to meet the input requirements of the Segformer model, expressed as:

[0083] ;

[0084] Among them, X represents the preprocessed image, which is used as the input of the model;

[0085] S22. Input the preprocessed image into the Segformer model. The mathematical expression is:

[0086] ;

[0087] S23. Use the Segformer algorithm to segment the image to generate a mask containing the hair area.

[0088] Further, step S3 includes the following steps:

[0089] S31, defining a hairstyle region as a region of interest;

[0090] S32, the boundary rectangle of the source image contour is expressed as:

[0091] ;

[0092] The bounding rectangle of the target image contour is expressed as:

[0093] ;

[0094] in, Indicates the coordinates of the upper left corner of the region of interest in the source image; Represents the width of the region of interest in the source image, that is, the number of pixels in the horizontal direction; Represents the height of the region of interest in the source image, that is, the number of pixels in the vertical direction; Indicates the coordinates of the upper left corner of the region of interest in the target image; Indicates the width of the region of interest in the target image; Indicates the height of the region of interest in the target image;

[0095] S33, the aligned image is calculated by the following formula

[0096] ;

[0097] in, Represents the scaled source image; Represents the scaling operation of an image; represents the original source image; Represents the width scaling factor, which is used to adjust the width of the source image to match the width of the target image; Represents the height scaling factor, which is used to adjust the height of the source image to match the height of the target image;

[0098] ;

[0099] in, Represents the adjusted source image; Translate represents the translation operation of the image; Indicates the horizontal translation of the source image, which is used to align the aligned image horizontally with the region of interest of the target image; Indicates the vertical translation of the source image, which is used to align the aligned image vertically with the region of interest of the target image.

[0100] Among them, the two formulas in S32 are used to describe the position and size of the region of interest (ROI, i.e. the hairstyle part) in the source image. Through these four parameters, we can accurately determine the position and size of the hairstyle in the source image;

[0101] For example, if we want to cut out the hairstyle part from the source image, we can crop it according to this rectangular frame. Suppose we have a portrait photo and want to extract the hairstyle part. Suppose we get (Unit is pixel), which means: the coordinate of the upper left corner of the hairstyle is (100,50), which is 100 pixels to the right and 50 pixels down from the upper left corner of the image; the width of the hairstyle is 200 pixels and the height is 300 pixels, so the position of the hairstyle area can be determined.

[0102] Further, step S4 includes the following steps:

[0103] S41. Calculate source image Color distribution histogram , and calculate the target image Color distribution histogram , where v represents the color value;

[0104] S42, calculating the cumulative distribution function of the source image and the target image;

[0105] S43. For each color value in the source image , find the color value closest to its cumulative distribution function in the target image , and Mapping , get the color-matched image .

[0106] Furthermore, in step S42, the cumulative distribution function of the source image is calculated by the following formula:

[0107] ;

[0108] The cumulative distribution function of the target image is calculated by the following formula:

[0109] .

[0110] Further, in step S43, the color value of the source image is Maps to the closest color value in the target image The process is expressed by the following formula:

[0111] ;

[0112] The color distribution of the source image is adjusted in the above manner to make it close to the color distribution of the target image.

[0113] In order to ensure the consistency of the hairstyle color with the target image, this scheme adopts color histogram matching technology. Color matching minimizes color differences by aligning the color histogram of the source image with the target image. The purpose of histogram matching is to make the color distribution of the source image as close as possible to the color distribution of the target image. This is achieved by adjusting the color value of each pixel of the source image so that its cumulative distribution function in the source image is aligned with the cumulative distribution function of the target image.

[0114] Furthermore, the specific steps of step S5 are as follows:

[0115] S51. In the generation phase of GAN, the model accepts the input source image, target image and related masks, and generates the fused image through the generator. ;

[0116] S52, evaluating the image through the discriminator D, which is used to distinguish the real image x from the generated image G(z);

[0117] S53, through the optimization process, gradually adjust the parameters of the generator G and the discriminator D to make the generated image close to the real effect;

[0118] S54. The above content is expressed by the following formula:

[0119] ;

[0120] in, represents the source image, A mask representing the hair, represents the target hairstyle image, and z represents random noise.

[0121] This formula means that by optimizing the generator G, a fusion image that is as realistic as possible is generated, so that the discriminator D cannot distinguish the difference between the generated fusion image and the real image. The goal of the generator is to minimize the discriminator's misjudgment of the generated image, while the goal of the discriminator is to maximize the accurate distinction between the real image and the generated image. When the discriminator cannot distinguish the difference between the generated fusion image and the real image, it means that the generated image is very realistic.

[0122] Furthermore, the optimization process in step S53 is as follows:

[0123] S531. Use gradient descent (Adam optimizer) to update the parameters of the generator and discriminator.

[0124] S532, the method of updating the discriminator parameters is as follows:

[0125] ;

[0126] in, represents the parameters of the discriminator; represents the learning rate; Represents the loss function of the discriminator About the discriminator parameters The gradient of

[0127] if is a positive number, which means that the current parameter increases the loss function, so the parameter value in this direction needs to be reduced; on the contrary, if the gradient is negative, the parameter value in this direction needs to be increased; in this way, the parameters of the discriminator D are constantly adjusted to find the parameter value that minimizes the loss function;

[0128] In each iteration, we calculate the gradient of the generator loss function and then update the parameters of the generator in the opposite direction of the gradient, with the goal of making the data generated by the generator look more like real data, thereby deceiving the discriminator D;

[0129] S533. The method of updating the generator parameters is as follows:

[0130] ;

[0131] in, Represents the parameters of the generator; Represents the loss function of the generator About generator parameters gradient.

[0132] The hairstyle fusion results generated by GAN are evaluated by the discriminator D to ensure the authenticity and naturalness of the generated images. During the training process, the parameters of the generator G and the discriminator D are continuously optimized to make the generated hairstyle images gradually approach the real effect.

[0133] Furthermore, the adversarial training process in step S1 is as follows:

[0134] S11. Generate a synthetic image that is consistent with the style of the input image through the generator, so that the generated image is as realistic as possible, so that it is difficult to distinguish from the real image. The loss function of the generator is expressed by the following formula:

[0135] ,

[0136] in, Represents the loss function of the generator. The purpose of the generator is to minimize the discriminator D on the generated image The recognition result makes Approaching 1, the generated image is close to the real image;

[0137] S12. The discriminator is used to distinguish between real images and generated images. The purpose is to maximize the accurate distinction between real images and generated images, that is, real images are judged as real as much as possible, and generated images are judged as false as much as possible. The loss function of the discriminator is expressed by the following formula:

[0138] ,

[0139] in, represents the loss function of the discriminator, represents the ability of the discriminator to correctly identify real images, Represents the ability of the discriminator to correctly identify generated images as fake;

[0140] S13. As training progresses, the generator gradually learns to imitate the details of real images, making the images generated by the generator increasingly difficult to distinguish from real images.

[0141] The loss function of the generator in the Generative Adversarial Network (GAN) is used to guide the generator to generate data that is closer to the real data distribution. The core idea of ​​GAN is to let the generator and the discriminator compete with each other through an adversarial process, so that the generator can generate sufficiently realistic data.

[0142] In GAN, the generator's loss function has the following main functions:

[0143] Measuring the gap between the data generated by the generator and the real data: The loss function evaluates the similarity between the data distribution produced by the generator and the real data distribution;

[0144] Optimize the weights of the generator: Through the gradient obtained by backpropagation of the loss function, the weights of the generator can be adjusted so that the generator can better simulate the distribution of real data.

[0145] The core goal of the loss function is to guide the generator G to generate high-quality samples that conform to the real data distribution. By continuously iteratively updating the parameters of the generator G, the generator G can eventually generate samples that are difficult for the discriminator D to distinguish from real data, that is, the images generated by the generator G become more and more realistic.

[0146] Indicates that the fake samples generated by the generator As the input of the discriminator D, the discriminator D evaluates it and outputs the result. During the training process, the goal of the generator is to make Close, because we want the discriminator to mistake the generated fake samples for real samples.

[0147] Through the above adversarial training and optimization process, the generator can generate more and more realistic images, while the discriminator continuously improves its identification ability, ultimately achieving high-quality image fusion.

[0148] Furthermore, step S6 includes adjusting the image details through image editing software to achieve the most natural visual effect.

[0149] The fusion results generated by GAN may still have inconsistent details, such as unrealistic hairstyle fit. If this happens, manual fine-tuning is required to ensure seamless fusion of the hairstyle and the head area, or repeat the generation of partial masks of the head and hair, and repeat the above operation process for multiple attempts.

[0150] Finally, the transparency of the fused image can be adjusted through Photoshop or other image editing software to achieve the most natural visual effect.

[0151] The above description is only an exemplary embodiment of the present invention, and does not limit the patent scope of the present invention. All equivalent structural changes made by using the contents of the present invention specification and drawings under the technical concept of the present invention, or directly / indirectly applied in other related technical fields are included in the patent protection scope of the present invention.

Claims

1. A hairstyle transplantation method based on generative adversarial network GAN, characterized in that: The following steps are involved: S1. Train the generative adversarial network model to make the generated images more realistic; S2, using image segmentation technology to generate masks of corresponding areas of the source image and the target image to ensure that only relevant areas are processed during the hair transplantation process; S3, scaling and moving the source image so that the hairstyle area of ​​the source image is aligned with the hairstyle area of ​​the target image; S4, adjusting the color distribution of the source image to make it close to the color distribution of the target image; S5, inputting the source image, the target image and the related mask into the model so that the model can fuse the hairstyle region of the target image with the head region of the source image, thereby realizing the hairstyle transplantation; S6, adjusting the details of the fused image; The step S2 comprises the following steps: S21, preprocess the image I through Processor to meet the input requirements of the Segformer model, expressed as: ； Among them, X represents the preprocessed image, which is used as the input of the model; S22. Input the preprocessed image into the Segformer model. The mathematical expression is: ； S23, segment the image using the Segformer algorithm to generate a mask containing the hair area; The step S3 comprises the following steps: S31, defining a hairstyle region as a region of interest; S32, the boundary rectangle of the source image contour is expressed as: ； The bounding rectangle of the target image contour is expressed as: ； in, Indicates the coordinates of the upper left corner of the region of interest in the source image; Represents the width of the region of interest in the source image, that is, the number of pixels in the horizontal direction; Represents the height of the region of interest in the source image, that is, the number of pixels in the vertical direction; Indicates the coordinates of the upper left corner of the region of interest in the target image; Indicates the width of the region of interest in the target image; Indicates the height of the region of interest in the target image; S33, the aligned image is calculated by the following formula ； in, Represents the scaled source image; Represents the scaling operation of an image; represents the original source image; Represents the width scaling factor, which is used to adjust the width of the source image to match the width of the target image; Represents the height scaling factor, which is used to adjust the height of the source image to match the height of the target image; ； in, Represents the adjusted source image; Translate represents the translation operation of the image; Indicates the horizontal translation of the source image, which is used to align the aligned image horizontally with the region of interest of the target image; Indicates the vertical translation of the source image, which is used to align the aligned image vertically with the region of interest of the target image; The step S4 comprises the following steps: S41. Calculate source image Color distribution histogram , and calculate the target image Color distribution histogram , where v represents the color value; S42, calculating the cumulative distribution function of the source image and the target image; S43. For each color value in the source image , find the color value closest to its cumulative distribution function in the target image , and Mapping , get the color-matched image .

2. The hairstyle transplantation method based on the generative adversarial network GAN according to claim 1 is characterized in that: In step S42, the cumulative distribution function of the source image is calculated by the following formula: ； The cumulative distribution function of the target image is calculated by the following formula: 。 3. The hairstyle transplantation method based on the generative adversarial network GAN according to claim 1, characterized in that: In step S43, the color value of the source image is Maps to the closest color value in the target image The process is expressed by the following formula: ； The color distribution of the source image is adjusted in the above manner to make it close to the color distribution of the target image.

4. The hairstyle transplantation method based on the generative adversarial network GAN according to claim 1, characterized in that: The specific steps of step S5 are as follows: S51. In the generation phase of GAN, the model accepts the input source image, target image and related masks, and generates the fused image through the generator. ; S52, evaluating the image through the discriminator D, which is used to distinguish the real image x from the generated image G(z); S53, through the optimization process, gradually adjust the parameters of the generator G and the discriminator D to make the generated image close to the real effect; S54. The above content is expressed by the following formula: ； in, represents the source image, A mask representing the hair, represents the target hairstyle image, and z represents random noise.

5. The hairstyle transplantation method based on the generative adversarial network GAN according to claim 4 is characterized in that: The optimization process in step S53 is as follows: S531, using gradient descent to update the parameters of the generator and the discriminator; S532, the method of updating the discriminator parameters is as follows: ； in, represents the parameters of the discriminator; represents the learning rate; Represents the loss function of the discriminator About the discriminator parameters The gradient of S533. The method of updating the generator parameters is as follows: ； in, Represents the parameters of the generator; Represents the loss function of the generator About generator parameters gradient.

6. The hairstyle transplantation method based on the generative adversarial network GAN according to claim 1, characterized in that: The adversarial training process in step S1 is as follows: S11. Generate a synthetic image that is consistent with the style of the input image through the generator, so that the generated image is as realistic as possible, so that it is difficult to distinguish from the real image. The loss function of the generator is expressed by the following formula: ， in, Represents the loss function of the generator. The purpose of the generator is to minimize the discriminator D on the generated image The recognition result makes Approaching 1, the generated image is close to the real image; S12. The discriminator is used to distinguish between real images and generated images. The purpose is to maximize the accurate distinction between real images and generated images, that is, real images are judged as real as much as possible, and generated images are judged as false as much as possible. The loss function of the discriminator is expressed by the following formula: ， in, represents the loss function of the discriminator, represents the ability of the discriminator to correctly identify real images, Represents the ability of the discriminator to correctly identify generated images as fake; S13. As training progresses, the generator gradually learns to imitate the details of real images, making the images generated by the generator increasingly difficult to distinguish from real images.

7. The hairstyle transplantation method based on the generative adversarial network GAN according to claim 1, characterized in that: The step S6 includes adjusting the picture details by using picture editing software to achieve the most natural visual effect.

Images