A face photo to sketch generation model training method, generation method, system and computer readable storage medium based on a diffusion model

Abstract

The application discloses a face photo to sketch generation model training method based on a diffusion model. The method comprises the following steps: initializing the diffusion model, obtaining a sample pair data set, selecting one sample pair as a training set and the rest as a test set; using the training set and a text instruction to perform image reconstruction on the diffusion model, and training the text instruction in the process; using the trained text instruction and the training set to input into the diffusion model to repeat the above text training process, until the training times reach a training times threshold, and then outputting the text instruction; using the diffusion model and the optimized text instruction to convert the face photo of the test set into a face sketch, and additional information can be embedded into the optimized text instruction in the test process. The application only uses one set of sketch sample to train the model, generates a sketch with high similarity to the original image, greatly accelerates the algorithm running speed, greatly saves the training cost, and is suitable for different painting styles.

Description

TECHNICAL FIELD

[0001] The present application belongs to the field of image processing of deep learning, and more particularly relates to a face photo to sketch generation model training method, generation method, system and computer readable storage medium based on a diffusion model. BACKGROUND

[0002] Face sketch is one of the most popular and basic portrait styles in the field of digital entertainment. However, the process of drawing a face sketch by an artist is time-consuming and laborious, which also leads to the fact that the size of the existing face sketch dataset is very small. Especially in today's big data era, the largest available face photo sketch dataset contains only a few thousand images. At the same time, due to the problem of artist painting style and level, the painting may be quite different from the actual face and difficult to identify. In many existing face photo sketch datasets, the sketch image is quite different from the original photo in terms of similarity. Considering the burden of creating face sketches manually, it is necessary to study the technology of automatically processing face photo to sketch. The existing face photo to sketch generation research mainly studies the work of directly converting photos to corresponding sketches, which can expand the existing face sketch dataset for future research.

[0003] Many face sketch generation works today, such as Peng et al. 2023 research work "HiFiSketch: High Fidelity Face Photo-Sketch Synthesis and Manipulation", have achieved good results. However, the generation ability of the methods used is still limited and requires a large amount of data for training. Although good generation effects can be achieved, it also means an increase in the corresponding training cost. For the sketch field, which requires a large amount of time and manpower cost to obtain a certain amount of training dataset, there are many burdens. At the same time, affected by the training dataset, although the generation effect can achieve similar painting quality and style as the training dataset, if the painting quality of the training dataset itself is limited, the sketch produced by the model is also difficult to maintain a high similarity with the original photo. Therefore, it is particularly crucial to explore a new method that can save training cost, improve efficiency, and have better recognition.

[0004] The patent application with the prior art publication number CN117745521A proposes an image style conversion method, device, electronic equipment and storage medium, which comprises: acquiring a to-be-processed image; encoding the to-be-processed image to obtain a first feature map; performing feature extraction on the to-be-processed image to obtain at least one second feature map, wherein each second feature map is used to represent one dominant feature in the to-be-processed image; and fusing the first feature map and the at least one second feature map to perform style conversion on the to-be-processed image to obtain a target image. This scheme has the problems of slow training speed and low drawing quality. SUMMARY

[0005] To overcome the problems of slow training speed, low drawing quality and the need for a large amount of data for training in the prior art, the present application provides a face photo to sketch generation model training method based on a diffusion model.

[0006] The first aspect of the present application provides a face photo to sketch generation model training method based on a diffusion model, comprising the following steps:

[0007] Initialize the time step of the pre-trained diffusion model, the training frequency threshold and the text instruction, obtain a sample pair data set composed of a face photo and its sketch, select one pair of samples as the training set and the rest as the test set;

[0008] Use the training set and the text instruction to perform an image reconstruction process on the pre-trained diffusion model, and use a preset strategy to train the text instruction during the process to obtain a trained text instruction;

[0009] Use the trained text instruction as input and input it into the diffusion model together with the training set to repeat the above text training process until the training frequency reaches the training frequency threshold, and output an optimized text instruction;

[0010] Use the diffusion model and the optimized text instruction to convert the face photo of the test set into a face sketch, and during the test process, additional information can be embedded into the optimized text instruction, and the additional information is marked as a part that needs special attention to more flexibly control the output effect.

[0011] Further, the image reconstruction process on the pre-trained diffusion model is performed without fine-tuning the model.

[0012] Further, the preset strategy for training the text instruction is a strategy combining the use of an image reconstruction strategy and the use of a CLIP determined editing direction, and the total loss function of the two strategies is combined with a gradient descent algorithm to train the text instruction. The formula of the gradient descent algorithm is:

[0013]

[0014] wherein γ represents a learning rate, is a loss function about C T .

[0015] Further, the image reconstruction strategy is specifically as follows:

[0016] encoding the sketch sample pair {x, y} to obtain a latent variable z x and z y , wherein x is a face image, and y is a sketch image corresponding to x;

[0017] z y is added with noise at the time step t, and the picture is gradually changed into a noisy picture;

[0018] the text instruction C T , z x , the time step t, and the noisy picture are input into a pre-trained denoising network to perform denoising at the time step t, and a denoised variable

[0019] In the reconstruction process, the text instruction is constantly optimized by a loss function, and the formula of the loss function is as follows:

[0020]

[0021] wherein ε is an encoder, ∈ is Gaussian noise, and ∈ θ is a denoising network.

[0022] Further, the strategy for training according to the editing direction determined by CLIP is specifically as follows:

[0023] by encoding the face photo and the sketch picture into an embedding space, the distance of the CLIP embedding between the pictures is calculated;

[0024] the distance is used to optimize the text instruction C T according to a loss function, the cosine similarity is selected as the loss function, and the formula of the loss function is as follows:

[0025]

[0026] wherein for a given training set sketch sample pair {x, y}, the distance of the CLIP embedding between x and y is calculated, denoted as Δ edit , that is, Δ edit = CLIP image (y) - CLIP image (x).

[0027] Further, the formula of the total loss function is:

[0028]

[0029] wherein, lambda rec and lambda edit are hyperparameters.

[0030] The second aspect of the present application provides a face photo to sketch generation method based on a diffusion model, which adopts the model trained above, uses the model to convert the picture to be converted to a sketch, and obtains the image after conversion to a sketch.

[0031] The third aspect of the present application provides a face photo to sketch generation model training system based on a diffusion model, which comprises a memory and a processor, and the memory comprises a face photo to sketch generation model training method based on a diffusion model, which, when executed by the processor, realizes the following steps:

[0032] The time step of the pre-trained diffusion model, the training frequency threshold and the text instruction are initialized, a sample pair data set composed of a face photo and its sketch is obtained, one of the sample pairs is selected as a training set, and the rest are selected as a test set;

[0033] The training set and the text instruction are used to perform an image reconstruction process on the pre-trained diffusion model, and a preset strategy is used to train the text instruction during the process to obtain a trained text instruction;

[0034] The trained text instruction is used as input and is input into the diffusion model together with the training set to repeat the above text training process until the training frequency reaches the training frequency threshold, and an optimized text instruction is output;

[0035] The diffusion model and the optimized text instruction are used to convert the face photo of the test set into a face sketch, and additional information can be embedded into the optimized text instruction during the test process, and the additional information is marked as a part that needs special attention to more flexibly control the output effect.

[0036] Further, the image reconstruction process using the training set on the pre-trained diffusion model is performed without fine-tuning the model.

[0037] The fourth aspect of the present application provides a computer readable storage medium, characterized in that the computer readable storage medium comprises a diffusion model-based face photo to sketch generation model training method program, and the diffusion model-based face photo to sketch generation model training method program is executed by a processor to realize the steps of the diffusion model-based face photo to sketch generation model training method.

[0038] Compared with the prior art, the beneficial effects of the technical scheme of the present application are:

[0039] The diffusion model is applied to the face sketch generation work, and the model is trained only with a set of sketch samples. Thanks to the powerful generation capability of the diffusion model, the generated sketch can overcome the drawing quality problem of the training data set itself, maintain a very high similarity with the original image, and be easy to identify. Compared with the previous face sketch generation technology, the present method greatly speeds up the operation, greatly saves the training cost, is efficient, and can adapt to different styles of drawing. BRIEF DESCRIPTION OF DRAWINGS

[0040] In order to make the purpose, technical scheme of the present application more clear, the present application provides the following drawings and makes the following description:

[0041] Figure 1 The method flowchart provided for the embodiment of the present application;

[0042] Figure 2 The diffusion model test schematic diagram provided for the embodiment of the present application;

[0043] Figure 3 The test effect on one-shotFace Sketch Dataset provided for the embodiment of the present application;

[0044] Figure 4 The generated sketch effect diagram with a gray picture provided for the embodiment of the present application;

[0045] Figure 5 The generated sketch effect diagram with rich texture provided for the embodiment of the present application;

[0046] Figure 6 The visual result diagram provided for the embodiment of the present application; DETAILED DESCRIPTION

[0047] In order to make the purpose, technical scheme of the present application more clear, the present application provides the following drawings and makes the following description:

[0048] In the following description, a large number of specific details are set forth in order to facilitate a thorough understanding of the application, but the application can also be practiced without other different ways from those described herein, therefore, the scope of protection of the application is not limited by the specific embodiments disclosed below.

[0049] The application provides a face photo to sketch generation model training method based on a diffusion model, as shown in Figure 1 The application provides a face photo to sketch generation model training method based on a diffusion model, as shown in

[0050] S1: initialize the time step of the pre-trained diffusion model, the training frequency threshold and the text instruction, obtain a face photo and its sketch portrait to form a sample pair data set, select one of the sample pairs as a training set, and the rest as a test set;

[0051] In one specific embodiment, a new data set named one-shot Face Sketch Dataset is formed by selecting face photos and their sketch portraits from CUFS, CUFSF data set and network sources. The new data set contains 300 face photo-sketch pairs, including different painting styles. Only one sample pair is used for training each time, and the rest is used for testing. The pre-trained diffusion model is used to optimize the instruction C T .

[0052] S2: using the training set and the text instruction, the image reconstruction process is performed on the pre-trained diffusion model, and the preset strategy is used to train the text instruction during the process to obtain the trained text instruction;

[0053] It should be noted that the image reconstruction process in step S2 on the pre-trained diffusion model is performed without fine-tuning the model.

[0054] More specifically, the preset strategy for training the text instruction in step S2 is a strategy that combines the image reconstruction strategy and the editing direction determined by CLIP. The text instruction is trained using the total loss function of the two strategies combined with the gradient descent algorithm. The formula of the gradient descent algorithm is:

[0055]

[0056] where γ represents the learning rate, is the loss function about the gradient of C T .

[0057] More specifically, the specific method of using the image reconstruction strategy is:

[0058] Encode the sketch sample pair {x, y} to obtain the latent variable z x and z y , where x is a face image and y is a sketch image corresponding to x;

[0059] Encode z y At time step t, add noise to the picture to gradually change it into a noisy picture;

[0060] Input the text instruction C T , z x , and the noisy picture into the pre-trained denoising network to perform denoising at time step t to obtain the denoised variable

[0061] During the reconstruction process, the text instruction is constantly optimized by a loss function, and the formula of the loss function is:

[0062]

[0063] where ε is an encoder, ∈ is Gaussian noise, and ∈ θ is a denoising network.

[0064] More specifically, the strategy for training using the editing direction determined by CLIP is as follows:

[0065] By encoding the face photo and the sketch picture into the embedding space, the distance between the CLIP embeddings of the pictures is calculated.

[0066] Using the distance, the text instruction C T is optimized according to a loss function, and the cosine similarity is selected as the loss function, and the formula of the loss function is:

[0067]

[0068] where, for a given training set sketch sample pair {x, y}, the distance between the CLIP embeddings of x and y is calculated, denoted as Δ edit , i.e. edit Δ image = CLIP image (y) - CLIP rec (x).

[0069] More specifically, the formula of the total loss function is:

[0070]

[0071] where λ edit and λ rec are hyperparameters.

[0072] In one specific embodiment, the AdamW optimizer is used, the learning rate γ = 0.001, λrec = 4, l edit = 0.1, in and optimization under the joint action of the text instructions C T , save the trained text instructions C T after training.

[0073] S3: use the trained text instructions as input, input into the diffusion model with the training set, repeat the above text training process until the training times reach the training times threshold, output the optimized text instructions;

[0074] S4: use the diffusion model and the optimized text instructions to convert the test set of face photos into face sketch images, as shown in Figure 3 Additional information can be embedded in the optimized text instructions during the test process, and the additional information is marked as a part that needs special attention to more flexibly control the output effect, as shown in Figure 2 .

[0075] Through the embedding of additional information, different styles of sketches can be controlled, as shown in Figure 4 The generated picture is a gray sketch effect diagram, and as shown in Figure 5 The generated picture is a sketch effect diagram with rich texture.

[0076] The second aspect of the present application provides a face photo to sketch generation method based on a diffusion model, which uses the above-mentioned trained model to convert the picture to be converted to sketch using the model, and obtains the converted sketch image.

[0077] The third aspect of the present application provides a face photo to sketch generation model training system based on a diffusion model, characterized in that the system comprises a memory and a processor, the memory comprises a face photo to sketch generation model training method based on a diffusion model, and the face photo to sketch generation model training method based on a diffusion model is executed by the processor to realize the following steps:

[0078] S1: initialize the time step of the pre-trained diffusion model, the training times threshold and the text instructions, obtain the face photo and its sketch image to form a sample pair data set, select one pair of samples as the training set, and the rest as the test set;

[0079] S2: use the training set and the text instructions to perform an image reconstruction process on the pre-trained diffusion model, use a preset strategy to train the text instructions during the process, and obtain the trained text instructions;

[0080] S3: using the trained text instruction as input, inputting into the diffusion model together with the training set to repeat the above text training process, until the training times reach the training times threshold, outputting the optimized text instruction;

[0081] S4: using the diffusion model and the optimized text instruction to convert the test set face photo into a human face sketch image, additional information can be embedded in the optimized text instruction during the test process, and the additional information is marked as a part that needs special attention to more flexibly control the output effect.

[0082] Further, the image reconstruction process using the training set on the pre-trained diffusion model is performed without fine-tuning the model.

[0083] The fourth aspect of the present application provides a computer readable storage medium, characterized in that the computer readable storage medium comprises a diffusion model based face photo to sketch generation model training method program, and the diffusion model based face photo to sketch generation model training method program is executed by a processor to realize the steps of the diffusion model based face photo to sketch generation model training method.

[0084] Table 1

[0085]

[0086] Table 1 is used to evaluate the generated sketch image and the original sketch photo by using the peak signal-to-noise ratio (PSNR), structural similarity index (SSIM), learning perceptual image block similarity (LPIPS), and Fréchet inception distance (FID) indicators. From the results, it can be found that the index evaluation of the sketch generated by the method of the present application is better than the original sketch, especially the structural similarity index SSIM is improved by several percentage points. Combined with the visual results, it can be determined that the method of the present application can generate a sketch with very high structural similarity to the original photo, as shown in Figure 6 .

[0087] Obviously, the above embodiments of the present application are only examples for clearly illustrating the present application, and are not intended to limit the embodiments of the present application. Based on the above description, other different forms of changes or variations can be made by those skilled in the art. Here, it is not necessary and impossible to exhaust all the embodiments. Any modification, equivalent replacement and improvement made within the spirit and principles of the present application shall be included in the protection scope of the claims of the present application.

Claims

1. A method for training a face photo to sketch generation model based on a diffusion model, characterized in that, The method comprises the following steps: initializing a time step of a pre-trained diffusion model, a training frequency threshold and a text instruction, obtaining a face photo and a sketch portrait to form a sample pair dataset, and using only one sample pair as a training set and the rest as a test set; using the training set and the text instruction to perform an image reconstruction process on the pre-trained diffusion model, training the text instruction using a preset strategy during the process, and obtaining a trained text instruction; using the trained text instruction as input and inputting the training set into the diffusion model to repeat the above text training process until the training frequency reaches the training frequency threshold, and outputting an optimized text instruction; using the diffusion model and the optimized text instruction to convert the face photo of the test set into a face sketch portrait, embedding additional information into the optimized text instruction during the test process, marking the additional information as a part that needs special attention to more flexibly control the output effect; wherein the preset strategy is a strategy combining an image reconstruction strategy and a CLIP determined editing direction, and a gradient descent algorithm is used to train the text instruction based on a total loss function of the two strategies, and the formula of the text instruction gradient descent algorithm is: In the formula, denotes the learning rate, is the loss function with respect to the gradient; the specific method of the image reconstruction strategy is: encoding a sketch sample pair {x, y} to obtain a latent variable and wherein x is a face image and y is a sketch image corresponding to x; To At time step t, the picture is progressively turned into a noisy picture by adding noise. Text instructions , At time step t, the noisy image is input into a pre-trained denoising network, and denoising is performed at time step t to obtain the denoised variables. ; the text instruction is continuously optimized by a loss function during the process, and the formula of the loss function is: wherein, is an encoder, is a Gaussian noise, is a denoising network.

2. The method of claim 1, wherein the method is based on a diffusion model. the image reconstruction process on the pre-trained diffusion model is performed without fine-tuning the model.

3. The method of claim 1, wherein the method is based on a diffusion model. the specific method of the strategy of training using the CLIP determined editing direction is: by encoding the face photo and the sketch picture into an embedding space, the distance of the CLIP embedding between the pictures is calculated; optimizing the text instructions according to a loss function using the distances Cosine similarity is selected as the loss function, and the formula of the loss function is: where, for a given training set sketch sample pair {x, y}, the distance between the CLIP embeddings of x and y, denoted as d(x, y), is computed as i.e. .

4. The method of claim 1, wherein the method is based on a diffusion model. the formula of the total loss function is: wherein and are hyperparameters.

5. A face photo to sketch generation method based on a diffusion model, the method uses the model trained in claims 1-4 to convert a picture to be converted into a sketch using the model to obtain a converted sketch image.

6. A face photo to sketch generation model training system based on a diffusion model, characterized in that, The system comprises a memory and a processor, and the memory comprises a diffusion model based face photo to sketch generation model training method program, which realizes the following steps when executed by the processor: initializing a time step of a pre-trained diffusion model, a training frequency threshold and a text instruction, obtaining a face photo and a sketch portrait to form a sample pair dataset, and using only one sample pair as a training set and the rest as a test set; using the training set and the text instruction to perform an image reconstruction process on the pre-trained diffusion model, training the text instruction using a preset strategy during the process, and obtaining a trained text instruction; using the trained text instruction as input and inputting the training set into the diffusion model to repeat the above text training process until the training frequency reaches the training frequency threshold, and outputting an optimized text instruction; using the diffusion model and the optimized text instruction to convert the face photo of the test set into a face sketch portrait, embedding additional information into the optimized text instruction during the test process, marking the additional information as a part that needs special attention to more flexibly control the output effect; The preset strategy is a combination of a training strategy using an image reconstruction strategy and an editing direction determined using CLIP, and a total loss function of the two strategies is combined with a gradient descent algorithm to train the text instruction, and a formula of the text instruction gradient descent algorithm is: In the formula, denotes the learning rate, is the loss function with respect to the gradient; The specific method of the image reconstruction strategy is: encoding a sketch sample pair {x, y} to obtain a latent variable and wherein x is a face image and y is a sketch image corresponding to x; To At time step t, the picture is progressively turned into a noisy picture by adding noise. Text instructions , At time step t, the noisy image is input into a pre-trained denoising network, and denoising is performed at time step t to obtain the denoised variables. ; The text instruction is continuously optimized by a loss function during the process, and a formula of the loss function is: wherein, is an encoder, is a Gaussian noise, is a denoising network.

7. The model training system for generating a sketch from a photo of a face based on a diffusion model according to claim 6, characterized in that, The image reconstruction process using the training set on the pre-trained diffusion model is performed without fine-tuning the model.

8. A computer-readable storage medium, characterized in that, The computer readable storage medium comprises a diffusion model-based face photo to sketch generation model training method program, and the diffusion model-based face photo to sketch generation model training method program is executed by the processor to realize the steps of the diffusion model-based face photo to sketch generation model training method according to any one of claims 1 to 4.

Images