Face generation and style migration method based on single photo

Abstract

The invention discloses a face generation and style migration method based on a single photo. The method comprises the following steps: S1, photo preprocessing: preprocessing a character photo input by a user; s2, training a generative model: training the generative model by using a large-scale face image data set to obtain a trained generative model; s3, style migration model training: using different styles of art work image data sets and a reference training style migration model to obtain a trained style migration model; s4, face generation: inputting the preprocessed image into the trained generative model to synthesize a new face image; and S5, style migration: fusing the synthesized new face image with the trained style migration model, and generating final face images with different style characteristics according to the artistic style selected by the user. The method has the advantages that on the basis of keeping the original face image content, a specific artistic style can be applied to the face image, and a new image with a unique artistic effect is created.

Description

Technical Field

[0001] The present invention relates to the technical field of computer vision and image processing, and in particular to a method for face generation and style transfer based on a single photo. Background Art

[0002] Facial images generated by existing technologies lack authenticity and naturalness, and may have unrealistic textures, blurred details or unnatural expressions; existing style transfer technologies are sometimes unable to accurately capture and transfer complex artistic styles, resulting in the generated image style being not pure enough or deviating from expectations; some existing technologies often generate results automatically, making it difficult for users to participate in and control the generation process, and are unable to meet personalized needs. Summary of the invention

[0003] The purpose of the present invention is to provide a method for face generation and style transfer based on a single photo, so as to solve the above-mentioned problems existing in the prior art.

[0004] In order to achieve the above object, the technical solution adopted by the present invention is as follows:

[0005] A face generation and style transfer method based on a single photo, comprising the following steps:

[0006] S1. Photo preprocessing: extracting facial geometric features of the person photo input by the user, and performing color correction and brightness adjustment operations on the person photo;

[0007] S2. Generative model training: Use a large-scale face image dataset to train the generative model and obtain a trained generative model;

[0008] S3, style transfer model training: Use image datasets of artworks of different styles and reference training style transfer models to obtain a trained style transfer model;

[0009] S4, face generation: input the preprocessed image into the trained generation model to synthesize a new face image;

[0010] S5. Style transfer: The synthesized new face image is merged with the trained style transfer model to generate a final face image with different style characteristics according to the artistic style selected by the user.

[0011] Preferably, step S1 specifically includes the following contents:

[0012] S11, obtaining a person photo input by a user;

[0013] S12, use ResNet network to extract potential facial features in the image;

[0014] S13, matching and comparing the extracted facial features with the pre-trained face detection model;

[0015] S14, based on the matching result, determining the position of the face in the image, and generating a rectangular bounding box surrounding the face;

[0016] S15, locating key points within the rectangular bounding box using key point annotation technology;

[0017] S16, using professional color space conversion and color balance algorithms to correct the color deviation of photos;

[0018] S17. Use brightness histogram equalization technology to optimize the brightness distribution of photos.

[0019] Preferably, step S2 specifically includes the following contents:

[0020] S21. Prepare a large-scale face image dataset and perform image cleaning, data enhancement and normalization.

[0021] S22. Construct a generative model based on the generative adversarial network GAN;

[0022] S23, training the generated model using the processed face dataset;

[0023] S24. During the training process, quantitative indicators are used to evaluate the quality and diversity of the images generated by the generative model, and the hyperparameters of the generative model are adjusted according to the evaluation results until the hyperparameters reach the optimal configuration to obtain a trained generative model.

[0024] Preferably, step S23 specifically comprises: constructing a generator, which is composed of a multi-layer neural network, starting from a random noise vector and gradually generating an image; then constructing a discriminator for distinguishing between real face images and images generated by the generator; and constructing a generation network based on the generator and the discriminator.

[0025] Preferably, step S24 is specifically that the generator and the discriminator are trained alternately. First, the discriminator is fixed, and the images generated by the generator are trained to be able to deceive the discriminator. Then, the generator is fixed, and the discriminator is trained to distinguish between real and generated images. This process is repeated until the generator can generate facial images that are realistic enough and difficult for the discriminator to distinguish.

[0026] Preferably, step S3 specifically includes the following contents:

[0027] S31. Collect a large number of image datasets of artworks in different styles and prepare a set of reference photos;

[0028] S32. Resize and crop the artwork image and reference photo so that they have the same specifications and format;

[0029] S33, extract style and content features of artwork images and reference photos using pre-trained convolutional networks;

[0030] S34, defining a loss function for the style transfer model, where the loss function is a weighted sum of content loss and style loss;

[0031] S35, extracting the style features and content features of the artwork image and the reference photo into the style transfer model and training it; during the training process, adjusting the parameters of the style transfer model by optimizing the loss function until the parameters reach the optimal configuration, thereby obtaining a trained style transfer model.

[0032] Preferably, step S4 specifically includes the following contents:

[0033] S41, using the trained generation model to perform feature extraction on the input preprocessed image;

[0034] S42, based on the knowledge and parameters obtained through training, the trained generative model transforms the extracted features and generates new features;

[0035] S43. Using the generated new features, the trained generative model gradually reconstructs the complete face image using upsampling operations.

[0036] Preferably, step S5 specifically includes the following contents:

[0037] S51, inputting the reconstructed face image into the trained style transfer model;

[0038] S52, the user determines the desired artistic style from the provided options, and the trained style transfer model extracts the corresponding style feature representation from its learned knowledge;

[0039] S53, the trained style transfer model performs feature extraction on the reconstructed face image to obtain content features;

[0040] S54, fusing the extracted content features of the face image with the selected artistic style features;

[0041] S55. Regenerate the final face image based on the fused features through the generation part of the trained style transfer model.

[0042] Preferably, the user selects a character photo and a desired target style using a human-computer interaction interface, and the human-computer interaction interface grants the user the authority to perform local fine adjustment and subtle optimization on the generated face image to meet the user's personalized creation needs; the human-computer interaction interface includes:

[0043] Photo selection module: users can select locally stored photos of people by browsing files or directly dragging and dropping;

[0044] Style selection menu: presents a variety of art style options to choose from, each with a clear name and sample image;

[0045] Image display area: used to display the generated face image in real time, with sufficient resolution and clarity so that users can clearly see the details;

[0046] Adjustment tools area: Provides a range of tools for local adjustments and fine-tuning.

[0047] Preferably, the final face image is output as a high-quality picture file, allowing users to download the picture or share it directly to social media platforms; user feedback is collected, and the generation model and style transfer model are continuously optimized.

[0048] The beneficial effects of the present invention are as follows: 1. The face images generated by the present invention have higher quality and naturalness, retaining the facial features of the original photo while generating more vivid and natural details. 2. The present invention can more accurately transfer the specified artistic style to the face image, accurately capture the complex artistic style, and make the generated image style more pure and more in line with expectations. 3. The present invention enhances user interactivity, and users can select their favorite reference photos and target styles, and make local adjustments and fine-tuning to the generated results, which can meet personalized needs. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] Figure 1 is a flow chart of a method in an embodiment of the present invention;

[0050] Figure 2 is a flow chart of preprocessing of a person photo in an embodiment of the present invention;

[0051] Figure 3 is a flow chart of generating model training in an embodiment of the present invention;

[0052] Figure 4 is a flow chart of style transfer model training according to an embodiment of the present invention;

[0053] Figure 5 4 is a flow chart of style transfer in an embodiment of the present invention. DETAILED DESCRIPTION

[0054] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings. It should be understood that the specific implementation methods described herein are only used to explain the present invention and are not used to limit the present invention.

[0055] like Figure 1As shown, in this embodiment, a method for face generation and style transfer based on a single photo is provided, including the following parts:

[0056] 1. Photo Preprocessing

[0057] The user inputs a photo of a person as a reference, and uses face detection algorithms and key point annotation technology to extract facial geometric features; pre-processing operations such as color correction and brightness adjustment are performed on the photo to provide high-quality input data for subsequent processing. Figure 2 As shown, it specifically includes the following contents:

[0058] 1.1. Image acquisition: Get the person photo input by the user.

[0059] 1.2. Feature extraction: Use the ResNet network to extract potential facial features in the image, such as skin color, contour, and the approximate shape of eyes, nose, and mouth.

[0060] 1.3. Model matching: Match and compare the extracted features with the pre-trained face detection model.

[0061] 1.4. Bounding box determination: Based on the matching results, determine the position of the face in the image and generate a rectangular bounding box surrounding the face.

[0062] 1.5. Locate the coordinates of key points: Use key point annotation technology to locate the key points within the rectangular bounding box, such as the inner and outer corners of the eyes, the tip of the nose, the corners of the mouth, etc., and record the coordinate positions of the key points in the image.

[0063] 1.6. Color correction: Use professional color space conversion and color balance algorithms to correct the color deviation of photos.

[0064] 1.7. Brightness adjustment: Use techniques such as brightness histogram equalization to optimize the brightness distribution of photos.

[0065] 2. Generative Model Training

[0066] Using large-scale face image datasets, we train generative models such as Generative Adversarial Networks (GANs) or Variational Autoencoders (VAEs) to learn detailed texture, skin color, expression, and other features of the face. Figure 3 As shown, it specifically includes the following contents:

[0067] 2.1. Data preprocessing: Prepare a large-scale face dataset, clean the images, and remove low-quality, blurred or incomplete face images; perform data enhancement and normalization on the image data to increase data diversity and reduce the risk of overfitting.

[0068] 2.2. Constructing a generative model: For the GAN network, first construct a generator, which consists of a multi-layer neural network, starting from a random noise vector and gradually generating images; then construct a discriminator to distinguish between real face images and images generated by the generator.

[0069] 2.3. Training model: The generator and discriminator are trained alternately. First, the discriminator is fixed and the images generated by the generator are trained to deceive the discriminator. Then the generator is fixed and the discriminator is trained to distinguish between real and generated images. This process is repeated until the generator can generate face images that are realistic enough and difficult for the discriminator to distinguish.

[0070] 2.4. Model evaluation and adjustment: Use some quantitative indicators, such as IS (Inception Score) and FID (French Inception Distance) to evaluate the quality and diversity of images, and adjust some hyperparameters of the model according to the evaluation results; the generative model with the optimal hyperparameter configuration is used as the trained generative model.

[0071] After sufficient training, the model can learn various features of the human face, such as skin texture, hair details, skin color, expressions, etc., and generate facial images with realistic skin color and delicate expression changes.

[0072] 3. Style transfer model training

[0073] Using a dataset of artwork images of different styles and reference photos, we train a style transfer model, which learns how to blend the content of the reference photo with the style of the artwork. Figure 4 As shown, it specifically includes the following contents:

[0074] 3.1. Collect datasets: Collect a large number of images of artworks in different styles, such as impressionism, abstraction, classicism, etc. At the same time, prepare a set of reference photos that will serve as the basis for the content.

[0075] 3.2. Data preprocessing: resize and crop the artwork images and reference photos to make them have uniform specifications and formats.

[0076] 3.3. Feature extraction: Use a pre-trained convolutional neural network to extract style features and content features of artwork images and reference photos.

[0077] 3.4. Define the loss function: The loss function consists of two parts, content loss and style loss. The total loss is the weighted sum of content loss and style loss.

[0078] 3.5. Training model: Adjust the parameters of the model by optimizing the loss function so that the generated image can preserve both the content of the reference photo and the style of the artwork. The style transfer model with the optimal parameter configuration is used as the trained style transfer model.

[0079] 4. Face Generation

[0080] Input the preprocessed person photo, and use the generative model trained in the second step to synthesize a new face image. The generated face should retain the facial features of the original photo, but have more vivid and natural details. Specifically, it includes the following contents:

[0081] 4.1. Feature extraction: In the second step, the trained generative model extracts features from the input reference photo. These features represent the facial feature information in the original photo.

[0082] 4.2. Generate new features: Based on the knowledge and parameters obtained through training, the model transforms the extracted features and generates new features. These new features are designed to increase details and naturalness while maintaining consistency with the original facial features.

[0083] 4.3. Image reconstruction: Using the generated new features, the model gradually reconstructs the complete face image through upsampling operations.

[0084] 5. Style Transfer

[0085] The face image generated in the fourth step is combined with the style transfer model trained in the third step to generate face images with different style characteristics according to the artistic style selected by the user. Figure 5 As shown, it specifically includes the following contents:

[0086] 5.1. Loading the model: Input the face image generated in step 4 into the style transfer model trained in step 3.

[0087] 5.2. Style feature extraction: The user determines the desired artistic style from the options provided, and the model extracts the corresponding style feature representation from its learned knowledge.

[0088] 5.3. Content feature extraction: Perform feature extraction on the generated face image again to obtain content features.

[0089] 5.4. Fusion: Fusion the content features of the extracted face image with the features of the selected artistic style.

[0090] 5.5. Image generation: Based on the fused features, the final face image is regenerated through the generation part of the model.

[0091] 6. Human-computer interaction and adjustment

[0092] Provides a friendly and intuitive human-computer interaction interface, allowing users to easily select reference photos and desired target styles. At the same time, users are given the authority to make local fine adjustments and subtle optimizations to the generated face images to meet their personalized creation needs. The interaction interface mainly includes the following modules.

[0093] (1) Photo selection module: Users can select reference photos stored locally by browsing files or directly dragging and dropping.

[0094] (2) Style selection menu: Present various art style options to choose from, such as a list, drop-down menu, or icon form. Each style has a clear name and example image to facilitate users’ intuitive understanding.

[0095] (3) Image display area: used to display the generated face image in real time, with sufficient resolution and clarity so that users can clearly see the details.

[0096] (4) Adjustment tool area: Provides a series of tools for local adjustment and fine-tuning.

[0097] 7. Image Output

[0098] Output the generated face images as high-quality image files, allowing users to download images or share them directly on social media platforms; collect user feedback and continuously optimize the generation model and style transfer model. Expand system functions to support more types of reference photos and artistic styles.

[0099] By adopting the above technical solution disclosed in the present invention, the following beneficial effects are obtained:

[0100] The present invention provides a method for face generation and style transfer based on a single photo. The face image generated by the present invention has higher quality and naturalness, retains the facial features of the original photo, and generates more vivid and natural details. The present invention can more accurately transfer a specified artistic style to a face image, accurately capture complex artistic styles, and make the generated image style more pure and more in line with expectations. The present invention enhances user interactivity, and users can choose their favorite reference photos and target styles, and make local adjustments and fine-tuning to the generated results, which can meet personalized needs.

[0101] The above is only a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principle of the present invention. These improvements and modifications should also be considered as the scope of protection of the present invention.

Claims

1. A face generation and style transfer method based on a single photo, characterized by: The following steps are included: S1. Photo preprocessing: extracting facial geometric features of the person photo input by the user, and performing color correction and brightness adjustment operations on the person photo; S2. Generative model training: Use a large-scale face image dataset to train the generative model and obtain a trained generative model; S3, style transfer model training: Use image datasets of artworks of different styles and reference training style transfer models to obtain a trained style transfer model; S4, face generation: input the preprocessed image into the trained generation model to synthesize a new face image; S5. Style transfer: The synthesized new face image is merged with the trained style transfer model to generate a final face image with different style characteristics according to the artistic style selected by the user.

2. The method for face generation and style transfer based on a single photo according to claim 1, characterized in that: Step S1 specifically includes the following contents: S11, obtaining a person photo input by a user; S12, use ResNet network to extract potential facial features in the image; S13, matching and comparing the extracted facial features with the pre-trained face detection model; S14, based on the matching result, determining the position of the face in the image, and generating a rectangular bounding box surrounding the face; S15, locating key points within the rectangular bounding box using key point annotation technology; S16, using professional color space conversion and color balance algorithms to correct the color deviation of photos; S17. Use brightness histogram equalization technology to optimize the brightness distribution of photos.

3. The method for face generation and style transfer based on a single photo according to claim 1, characterized in that: Step S2 specifically includes the following contents: S21. Prepare a large-scale face image dataset and perform image cleaning, data enhancement and normalization. S22. Construct a generative model based on the generative adversarial network GAN; S23, training the generated model using the processed face dataset; S24. During the training process, quantitative indicators are used to evaluate the quality and diversity of the images generated by the generative model, and the hyperparameters of the generative model are adjusted according to the evaluation results until the hyperparameters reach the optimal configuration to obtain a trained generative model.

4. The method for face generation and style transfer based on a single photo according to claim 3, characterized in that: Step S23 specifically includes constructing a generator, which is composed of a multi-layer neural network, starting from a random noise vector and gradually generating an image; then constructing a discriminator for distinguishing between a real face image and an image generated by the generator; Construct a generation network based on the generator and the discriminator.

5. The method for face generation and style transfer based on a single photo according to claim 3, characterized in that: Specifically, step S24 is to alternately train the generator and the discriminator. First, the discriminator is fixed, and the images generated by the generator are trained to be able to deceive the discriminator. Then, the generator is fixed, and the discriminator is trained to distinguish between real and generated images. This process is repeated until the generator can generate face images that are realistic enough and difficult for the discriminator to distinguish.

6. The method for face generation and style transfer based on a single photo according to claim 1, characterized in that: Step S3 specifically includes the following contents: S31. Collect a large number of image datasets of artworks in different styles and prepare a set of reference photos; S32. Resize and crop the artwork image and reference photo so that they have the same specifications and format; S33, extract style and content features of artwork images and reference photos using pre-trained convolutional networks; S34, defining a loss function for the style transfer model, where the loss function is a weighted sum of content loss and style loss; S35, extracting style features and content features of the artwork image and the reference photo into a style transfer model to train the model; During the training process, the parameters of the style transfer model are adjusted by optimizing the loss function until the parameters reach the optimal configuration to obtain the trained style transfer model.

7. The method for face generation and style transfer based on a single photo according to claim 1, characterized in that: Step S4 specifically includes the following contents: S41, using the trained generation model to perform feature extraction on the input preprocessed image; S42, based on the knowledge and parameters obtained through training, the trained generative model transforms the extracted features and generates new features; S43. Using the generated new features, the trained generative model gradually reconstructs the complete face image using upsampling operations.

8. The method for face generation and style transfer based on a single photo according to claim 7, characterized in that: Step S5 specifically includes the following contents: S51, inputting the reconstructed face image into the trained style transfer model; S52, the user determines the desired artistic style from the provided options, and the trained style transfer model extracts the corresponding style feature representation from its learned knowledge; S53, the trained style transfer model performs feature extraction on the reconstructed face image to obtain content features; S54, fusing the extracted content features of the face image with the selected artistic style features; S55. Regenerate the final face image based on the fused features through the generation part of the trained style transfer model.

9. The method for face generation and style transfer based on a single photo according to any one of claims 1 to 8, characterized in that: The user uses the human-computer interaction interface to select a person's photo and a desired target style. The human-computer interaction interface gives the user the authority to make local fine adjustments and subtle optimizations to the generated face image to meet the user's personalized creation needs; the human-computer interaction interface includes: Photo selection module: users can select locally stored photos of people by browsing files or directly dragging and dropping; Style selection menu: presents a variety of art style options to choose from, each with a clear name and sample image; Image display area: used to display the generated face image in real time, with sufficient resolution and clarity so that users can clearly see the details; Adjustment tools area: Provides a range of tools for local adjustments and fine-tuning.

10. The method for face generation and style transfer based on a single photo according to any one of claims 1 to 8, characterized in that: Output the final face image as a high-quality picture file, allowing users to download the image or share it directly on social media platforms; Collect user feedback and continuously optimize the generation model and style transfer model.

Images