Face photo-to-sketch generation model training method, generation method and system based on diffusion model, and computer readable storage medium

Abstract

The invention discloses a face photo-to-sketch generation model training method based on a diffusion model. Comprising the following steps: initializing a diffusion model, obtaining a sample pair data set, selecting one pair of samples as a training set, and taking the rest as a test set; performing image reconstruction on the diffusion model by using the training set and the text instruction, and training the text instruction in the process; inputting the trained text instruction and the training set into a diffusion model together to repeat the text training process until the training frequency reaches a training frequency threshold value, and outputting the text instruction; face photos in a test set are converted into face sketch images through the diffusion model and the optimized text instruction, and extra information can be embedded into the optimized text instruction in the test process. According to the method, only one group of sketch samples are used for training the model, the generated sketch keeps extremely high similarity with the original image, the algorithm running speed is greatly increased, the training cost is greatly saved, and the method can adapt to painting of different styles.

Description

Technical Field

[0001] The present invention belongs to the field of image processing of deep learning, and more specifically, to a facial photo-to-sketch generation model training method, generation method, system and computer-readable storage medium based on a diffusion model. Background Art

[0002] Face sketching is one of the most popular and basic portrait styles in the field of digital entertainment. However, the process of drawing facial sketches by artists is time-consuming and laborious, which has led to the small size of existing face sketch datasets. Especially in today's big data era, the largest available face photo sketch dataset contains only thousands of images. At the same time, due to the artist's painting style and level, the paintings may be very different from the actual face and difficult to identify. In many existing face photo sketch datasets, there is a certain gap in the similarity between the sketched image and the original photo. Considering the burden of manually creating face sketches, it is necessary to study the technology of automatically processing face photos to sketches. The existing research work on face photo to sketch generation mainly studies the work of directly converting photos into corresponding sketches, which can expand the existing face sketch dataset for future research.

[0003] Nowadays, many face sketch generation works, such as Peng et al.'s 2023 research work "HiFiSketch: High Fidelity Face Photo-Sketch Synthesis and Manipulation", have achieved good results. However, the generation capacity of the method used is still limited, and a large amount of data is required for training. Although good generation effects can be obtained, it also means an increase in the corresponding training costs. For the field of sketching, which requires a lot of time and manpower to obtain a certain number of training data sets, there are even more burdens. At the same time, affected by the training data set, although the generation effect can maintain a similar painting quality and style to the training data set, if the painting quality of the training data set itself is limited, the sketch generated by the model is also difficult to maintain a high degree of similarity with the original photo. Therefore, it is particularly important to explore new methods that can save training costs, improve efficiency, and have better recognition.

[0004] The invention patent with the prior art publication number CN117745521A proposes an image style conversion method, device, electronic device and storage medium, the method comprising: obtaining an image to be processed; encoding the image to be processed to obtain a first feature map; extracting features from the image to be processed to obtain at least one second feature map, wherein each second feature map is used to characterize a dominant feature in the image to be processed; fusing the first feature map with the at least one second feature map to perform style conversion on the image to be processed to obtain a target image. This solution has the problems of slow training speed and low drawing quality. Summary of the invention

[0005] In order to overcome the problems in the prior art of slow training speed, low drawing quality and need for a large amount of data for training, the present invention provides a method for training a face photo to sketch generation model based on a diffusion model.

[0006] The first aspect of the present invention provides a method for training a face photo to sketch generation model based on a diffusion model, comprising the following steps:

[0007] Initialize the time step, training number threshold and text instructions of the pre-trained diffusion model, obtain face photos and their sketches to form a sample pair data set, select one pair of samples as the training set, and the rest as the test set;

[0008] Performing an image reconstruction process on a pre-trained diffusion model using the training set and the text instructions, during which a preset strategy is used to train the text instructions to obtain trained text instructions;

[0009] Using the trained text instructions as input, inputting them together with the training set into the diffusion model to repeat the above text training process until the number of training times reaches a training times threshold, and outputting optimized text instructions;

[0010] The face photos of the test set are converted into face sketch portraits using the diffusion model and the optimized text instructions. During the test, additional information can be embedded into the optimized text instructions, and the additional information can be marked as a part that requires special attention to more flexibly control the output effect.

[0011] Furthermore, the image reconstruction process based on the pre-trained diffusion model is performed without fine-tuning the model.

[0012] Furthermore, the preset strategy for training text instructions is to combine the strategy of training by using the image reconstruction strategy and the strategy of training by using the editing direction determined by CLIP, and train the text instructions by using the total loss function of the two strategies combined with the gradient descent algorithm. The formula of the gradient descent algorithm is:

[0013]

[0014] Among them, γ represents the learning rate, is the loss function About C T gradient.

[0015] Furthermore, the specific method of adopting the image reconstruction strategy is:

[0016] Encode the sketch sample pair {x, y} to obtain the latent variable z x and z y , where x is the face image and y is the sketch image corresponding to x;

[0017] Right y Add noise at time step t, and gradually turn the image into a noisy image;

[0018] The text instruction C T , z x , time step t, the noise image is input into the pre-trained denoising network, denoised at time step t, and the denoised variable is obtained

[0019] During the reconstruction process, the text instructions are continuously optimized through the loss function. The formula of the loss function is:

[0020]

[0021] Among them, ε is the encoder, ∈ is Gaussian noise, ∈ θ It is a denoising network.

[0022] Furthermore, the specific method of the strategy for training using the editing direction determined by CLIP is:

[0023] By encoding the face photo and the sketch image into the embedding space, the distance between the CLIP embeddings of the images is calculated;

[0024] Utilize the distance to optimize the text instruction C according to the loss function T , cosine similarity is selected as the loss function, and the formula of the loss function is:

[0025]

[0026] For a given pair of training set sketch samples {x, y}, the distance between the CLIP embeddings of x and y is calculated, denoted as Δ edit , that is, Δ edit =CLIP image (y)-CLIP image (x).

[0027] Furthermore, the formula of the total loss function is:

[0028]

[0029] Among them, λ rec and λ edit is a hyperparameter.

[0030] The second aspect of the present invention provides a method for generating a face photo-to-sketch conversion based on a diffusion model. The method adopts the model obtained through the above training, and utilizes the model to convert a picture to be converted into a sketch to obtain an image after the conversion into a sketch.

[0031] The third aspect of the present invention provides a facial photo-to-sketch generation model training system based on a diffusion model, characterized in that the system comprises: a memory and a processor, wherein the memory comprises a facial photo-to-sketch generation model training method program based on a diffusion model, and when the facial photo-to-sketch generation model training method program based on a diffusion model is executed by the processor, the following steps are implemented:

[0032] Initialize the time step, training number threshold and text instructions of the pre-trained diffusion model, obtain face photos and their sketches to form a sample pair data set, select one pair of samples as the training set, and the rest as the test set;

[0033] Performing an image reconstruction process on a pre-trained diffusion model using the training set and the text instructions, during which a preset strategy is used to train the text instructions to obtain trained text instructions;

[0034] Using the trained text instructions as input, inputting them together with the training set into the diffusion model to repeat the above text training process until the number of training times reaches a training times threshold, and outputting optimized text instructions;

[0035] The face photos of the test set are converted into face sketch portraits using the diffusion model and the optimized text instructions. During the test, additional information can be embedded into the optimized text instructions, and the additional information can be marked as a part that requires special attention to more flexibly control the output effect.

[0036] Furthermore, the image reconstruction process using the training set on the pre-trained diffusion model is performed without fine-tuning the model.

[0037] The fourth aspect of the present invention provides a computer-readable storage medium, characterized in that the computer-readable storage medium includes a facial photo-to-sketch generation model training method program based on a diffusion model, and when the facial photo-to-sketch generation model training method program based on a diffusion model is executed by a processor, the steps of the facial photo-to-sketch generation model training method based on a diffusion model are implemented.

[0038] Compared with the prior art, the technical solution of the present invention has the following beneficial effects:

[0039] The diffusion model is applied to the generation of face sketches. Only one set of sketch samples is used to train the model. Thanks to the powerful generation ability of the diffusion model, the generated sketches can overcome the painting quality problems of the training data set itself, maintain a very high similarity with the original image, and are easy to identify. Compared with previous face sketch generation technologies, this method runs much faster, greatly saves training costs, is highly efficient, and can adapt to different styles of painting. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] In order to make the purpose and technical solution of the present invention clearer, the present invention provides the following drawings and descriptions:

[0041] Figure 1 A flow chart of a method provided by an embodiment of the present invention;

[0042] Figure 2 A schematic diagram of a diffusion model test provided by an embodiment of the present invention;

[0043] Figure 3 The test results of the embodiment of the present invention on the one-shot Face Sketch Dataset;

[0044] Figure 4 A sketch effect diagram with a grayish picture generated by an embodiment of the present invention;

[0045] Figure 5 A realistic sketch effect diagram with rich textures generated by an embodiment of the present invention;

[0046] Figure 6 A visualization result diagram provided by an embodiment of the present invention; DETAILED DESCRIPTION

[0047] In order to more clearly understand the above-mentioned purpose, features and advantages of the present invention, the present invention is further described in detail below in conjunction with the accompanying drawings and specific embodiments. It should be noted that the embodiments of the present application and the features in the embodiments can be combined with each other without conflict.

[0048] In the following description, many specific details are set forth to facilitate a full understanding of the present invention. However, the present invention may also be implemented in other ways different from those described herein. Therefore, the protection scope of the present invention is not limited to the specific embodiments disclosed below.

[0049] The present invention provides a method for training a face photo-to-sketch generation model based on a diffusion model. Figure 1 The figure shows a method flow chart of a method for training a face photo to sketch generation model based on a diffusion model. The specific steps are as follows:

[0050] S1: Initialize the time step, training number threshold and text instructions of the pre-trained diffusion model, obtain face photos and their sketches to form a sample pair data set, select one pair of samples as the training set, and the rest as the test set;

[0051] In a specific embodiment, a new dataset named one-shot Face Sketch Dataset is selected from CUFS, CUFSF datasets and face photos and sketches from the Internet. It contains 300 face photo-sketch pairs, including different painting styles. Only one pair of samples is used for each training, and the rest are used for testing. The frozen pre-trained diffusion model is used to optimize the instruction C under the training number threshold N = 1000 steps and time step t = 1000. T .

[0052] S2: performing an image reconstruction process on a pre-trained diffusion model using the training set and the text instructions, during which a preset strategy is used to train the text instructions to obtain trained text instructions;

[0053] It should be noted that the image reconstruction process on the pre-trained diffusion model in step S2 is performed without fine-tuning the model.

[0054] More specifically, the preset strategy for training text instructions in step S2 is to combine the strategy of training using the image reconstruction strategy and the strategy of training using the editing direction determined by CLIP, and train the text instructions using the total loss function of the two strategies combined with the gradient descent algorithm. The formula of the gradient descent algorithm is:

[0055]

[0056] Among them, γ represents the learning rate, is the loss function About C T gradient.

[0057] More specifically, the specific method of adopting the image reconstruction strategy is:

[0058] Encode the sketch sample pair {x, y} to obtain the latent variable z x and z y , where x is the face image and y is the sketch image corresponding to x;

[0059] Right y Add noise at time step t, and gradually turn the image into a noisy image;

[0060] The text instruction C T , z x , time step t, the noise image is input into the pre-trained denoising network, denoised at time step t, and the denoised variable is obtained

[0061] During the reconstruction process, the text instructions are continuously optimized through the loss function. The formula of the loss function is:

[0062]

[0063] Among them, ε is the encoder, ∈ is Gaussian noise, ∈ θ It is a denoising network.

[0064] More specifically, the strategy for training using the editing direction determined by CLIP is as follows:

[0065] By encoding the face photo and the sketch image into the embedding space, the distance between the CLIP embeddings of the images is calculated;

[0066] Utilize the distance to optimize the text instruction C according to the loss function T , cosine similarity is selected as the loss function, and the formula of the loss function is:

[0067]

[0068] For a given pair of training set sketch samples {x, y}, the distance between the CLIP embeddings of x and y is calculated, denoted as Δ edit , that is, Δ edit =CLIP image (y)-CLIP image (x).

[0069] More specifically, the formula of the total loss function is:

[0070]

[0071] Among them, λ rec and λ edit is a hyperparameter.

[0072] In a specific embodiment, the AdamW optimizer is used with a learning rate of γ = 0.001, λrec =4,λ edit =0.1, in and The text instruction C T Optimize and save the text instructions C obtained after training. T .

[0073] S3: using the trained text instructions as input, inputting them together with the training set into the diffusion model to repeat the above text training process, until the number of training times reaches the training times threshold, outputting the optimized text instructions;

[0074] S4: Use the diffusion model and the optimized text instructions to convert the face photos of the test set into face sketch portraits, such as Figure 3 As shown, during the test process, additional information can be embedded into the optimized text instructions, and the additional information can be marked as a part that requires special attention to control the output effect more flexibly, such as Figure 2 shown.

[0075] By embedding additional information, we can control the generation of sketches of different styles, such as Figure 4 The following is a gray sketch of the generated image. Figure 5 Shown is the resulting texture-rich, realistic sketch.

[0076] The second aspect of the present invention provides a method for generating a face photo-to-sketch conversion based on a diffusion model. The method adopts the model obtained through the above training, and utilizes the model to convert a picture to be converted into a sketch to obtain an image after the conversion into a sketch.

[0077] The third aspect of the present invention provides a facial photo-to-sketch generation model training system based on a diffusion model, characterized in that the system comprises: a memory and a processor, wherein the memory comprises a facial photo-to-sketch generation model training method program based on a diffusion model, and when the facial photo-to-sketch generation model training method program based on a diffusion model is executed by the processor, the following steps are implemented:

[0078] S1: Initialize the time step, training number threshold and text instructions of the pre-trained diffusion model, obtain face photos and their sketches to form a sample pair data set, select one pair of samples as the training set, and the rest as the test set;

[0079] S2: performing an image reconstruction process on a pre-trained diffusion model using the training set and the text instructions, during which a preset strategy is used to train the text instructions to obtain trained text instructions;

[0080] S3: using the trained text instructions as input, inputting them together with the training set into the diffusion model to repeat the above text training process, until the number of training times reaches the training times threshold, outputting the optimized text instructions;

[0081] S4: The face photos of the test set are converted into face sketches using the diffusion model and the optimized text instructions. During the test, additional information can be embedded into the optimized text instructions, and the additional information can be marked as a part that requires special attention to control the output effect more flexibly.

[0082] Furthermore, the image reconstruction process using the training set on the pre-trained diffusion model is performed without fine-tuning the model.

[0083] The fourth aspect of the present invention provides a computer-readable storage medium, characterized in that the computer-readable storage medium includes a facial photo-to-sketch generation model training method program based on a diffusion model, and when the facial photo-to-sketch generation model training method program based on a diffusion model is executed by a processor, the steps of the facial photo-to-sketch generation model training method based on a diffusion model are implemented.

[0084] Table 1

[0085]

[0086] Table 1 shows the evaluation of the generated sketch image and the original corresponding sketch photo using peak signal-to-noise ratio (PSNR), structural similarity index (SSIM), learning perceptual image block similarity (LPIPS), and Fréchet initiation distance (FID). From the results, it can be found that the index evaluation of the sketch generated by the method proposed in the present invention is better than the original sketch, especially the structural similarity index SSIM is improved by more than ten percentage points. Combined with the visualization results, it can be concluded that the method of the present invention can generate a sketch with a very high structural similarity to the original photo, such as Figure 6 shown.

[0087] Obviously, the above embodiments of the present invention are merely examples for clearly illustrating the present invention, and are not intended to limit the embodiments of the present invention. For those skilled in the art, other different forms of changes or modifications can be made based on the above description. It is not necessary and impossible to list all the embodiments here. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present invention should be included in the protection scope of the claims of the present invention.

Claims

1. A method for training a face photo to sketch generation model based on a diffusion model, characterized in that: The steps include: Initialize the time step, training number threshold and text instructions of the pre-trained diffusion model, obtain face photos and their sketches to form a sample pair data set, select one pair of samples as the training set, and the rest as the test set; Performing an image reconstruction process on a pre-trained diffusion model using the training set and the text instructions, during which a preset strategy is used to train the text instructions to obtain trained text instructions; Using the trained text instructions as input, inputting them together with the training set into the diffusion model to repeat the above text training process until the number of training times reaches a training times threshold, and outputting optimized text instructions; The face photos of the test set are converted into face sketch portraits using the diffusion model and the optimized text instructions. During the test, additional information can be embedded into the optimized text instructions, and the additional information can be marked as a part that requires special attention to more flexibly control the output effect.

2. According to the method for training a face photo to sketch generation model based on a diffusion model in claim 1, it is characterized in that: The image reconstruction process based on the pre-trained diffusion model is performed without fine-tuning the model.

3. The method for training a facial photo-to-sketch generation model based on a diffusion model according to claim 1, characterized in that: The preset strategy for training text instructions is to combine the strategy of training by using the image reconstruction strategy and the strategy of training by using the editing direction determined by CLIP, and train the text instructions by using the total loss function of the two strategies combined with the gradient descent algorithm. The formula of the gradient descent algorithm is: Among them, γ represents the learning rate, is the loss function About C T gradient.

4. The method for training a facial photo-to-sketch generation model based on a diffusion model according to claim 3, characterized in that: The specific method of adopting the image reconstruction strategy is: Encode the sketch sample pair {x, y} to obtain the latent variable z x and z y , where x is the face image and y is the sketch image corresponding to x; Right y Add noise at time step t, and gradually turn the image into a noisy image; The text instruction C T , z x , time step t, the noise image is input into the pre-trained denoising network, denoised at time step t, and the denoised variable is obtained In the process, the text instructions are continuously optimized through the loss function, and the formula of the loss function is: Among them, ε is the encoder, ∈ is Gaussian noise, ∈ θ It is a denoising network.

5. The method for training a facial photo-to-sketch generation model based on a diffusion model according to claim 3, characterized in that: The specific method of the strategy for training using the editing direction determined by CLIP is: By encoding the face photo and the sketch image into the embedding space, the distance between the CLIP embeddings of the images is calculated; Utilize the distance to optimize the text instruction C according to the loss function T , cosine similarity is selected as the loss function, and the formula of the loss function is: For a given pair of training set sketch samples {x, y}, the distance between the CLIP embeddings of x and y is calculated, denoted as Δ edit , that is, Δ edit =CLIP image (y)-CLIP image (x).

6. The method for training a facial photo-to-sketch generation model based on a diffusion model according to claim 3, characterized in that: The formula of the total loss function is: Among them, λ rec and λ edit is a hyperparameter.

7. A method for generating a face photo-to-sketch conversion based on a diffusion model, the method adopting the model trained according to claims 1-6, utilizing the model to convert a picture to be converted into a sketch, and obtaining an image after the conversion into a sketch.

8. A facial photo to sketch generation model training system based on diffusion model, characterized in that: The system includes: a memory and a processor, wherein the memory includes a training method program for converting a face photo into a sketch generation model based on a diffusion model, and when the training method program for converting a face photo into a sketch generation model based on a diffusion model is executed by the processor, the following steps are implemented: Initialize the time step, training number threshold and text instructions of the pre-trained diffusion model, obtain face photos and their sketches to form a sample pair data set, select one pair of samples as the training set, and the rest as the test set; Performing an image reconstruction process on a pre-trained diffusion model using the training set and the text instructions, during which a preset strategy is used to train the text instructions to obtain trained text instructions; Using the trained text instructions as input, inputting them together with the training set into the diffusion model to repeat the above text training process until the number of training times reaches a training times threshold, and outputting optimized text instructions; The face photos of the test set are converted into face sketch portraits using the diffusion model and the optimized text instructions. During the test, additional information can be embedded into the optimized text instructions, and the additional information can be marked as a part that requires special attention to more flexibly control the output effect.

9. The facial photo to sketch generation model training system based on diffusion model according to claim 8, characterized in that: The image reconstruction process using the training set on the pre-trained diffusion model is performed without fine-tuning the model.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium includes a facial photo-to-sketch generation model training method program based on a diffusion model. When the facial photo-to-sketch generation model training method program based on a diffusion model is executed by a processor, the steps of a facial photo-to-sketch generation model training method based on a diffusion model as described in any one of claims 1 to 6 are implemented.

Images